2022-12-01T10:12:06.2105579Z Requested labels: linux.8xlarge.nvidia.gpu
2022-12-01T10:12:06.2105655Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/89997/merge
2022-12-01T10:12:06.2105679Z Waiting for a runner to pick up this job...
2022-12-01T10:12:06.9375761Z Job is about to start running on the runner: i-0d2a7b5e684093f31 (organization)
2022-12-01T10:12:11.3116337Z Current runner version: '2.299.1'
2022-12-01T10:12:11.3123834Z Runner name: 'i-0d2a7b5e684093f31'
2022-12-01T10:12:11.3124464Z Runner group name: 'Default'
2022-12-01T10:12:11.3125288Z Machine name: 'ip-10-0-3-202'
2022-12-01T10:12:11.3127939Z ##[group]GITHUB_TOKEN Permissions
2022-12-01T10:12:11.3128895Z Actions: read
2022-12-01T10:12:11.3129257Z Checks: read
2022-12-01T10:12:11.3129633Z Contents: read
2022-12-01T10:12:11.3130025Z Deployments: read
2022-12-01T10:12:11.3130372Z Discussions: read
2022-12-01T10:12:11.3130805Z Issues: read
2022-12-01T10:12:11.3131184Z Metadata: read
2022-12-01T10:12:11.3131523Z Packages: read
2022-12-01T10:12:11.3131945Z Pages: read
2022-12-01T10:12:11.3132349Z PullRequests: read
2022-12-01T10:12:11.3132786Z RepositoryProjects: read
2022-12-01T10:12:11.3133230Z SecurityEvents: read
2022-12-01T10:12:11.3133629Z Statuses: read
2022-12-01T10:12:11.3133965Z ##[endgroup]
2022-12-01T10:12:11.3138162Z Secret source: None
2022-12-01T10:12:11.3139140Z Prepare workflow directory
2022-12-01T10:12:11.6518844Z Prepare all required actions
2022-12-01T10:12:11.6761491Z Getting action download info
2022-12-01T10:12:11.9020375Z Download action repository 'pytorch/pytorch@master' (SHA:850b53bbee82fb194af85b566aedee94b96def32)
2022-12-01T10:12:15.2143793Z Download action repository 'pytorch/test-infra@main' (SHA:1f415583bdcd967e33ea8fd05be71ed0bdf19880)
2022-12-01T10:12:15.4532796Z Download action repository 'nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767' (SHA:7d4a37704547a311dbb66ebdf5b23ec19374a767)
2022-12-01T10:12:15.5866244Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2022-12-01T10:12:16.1053463Z Getting action download info
2022-12-01T10:12:16.3103874Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2022-12-01T10:12:16.5181826Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2022-12-01T10:12:16.5184325Z ##[group] Inputs
2022-12-01T10:12:16.5184837Z   build-environment: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:12:16.5186227Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
]}

2022-12-01T10:12:16.5187620Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:12:16.5188122Z   sync-tag: 
2022-12-01T10:12:16.5188445Z ##[endgroup]
2022-12-01T10:12:16.5189746Z Complete job name: linux-bionic-cuda11.6-py3.10-gcc7 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu)
2022-12-01T10:12:16.6324820Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2022-12-01T10:12:16.6325213Z with:
2022-12-01T10:12:16.6325482Z   submodules: recursive
2022-12-01T10:12:16.6325744Z   fetch-depth: 0
2022-12-01T10:12:16.6325966Z env:
2022-12-01T10:12:16.6326211Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:12:16.6326481Z ##[endgroup]
2022-12-01T10:12:16.6638130Z ##[group]Run retry () {
2022-12-01T10:12:16.6638469Z retry () {
2022-12-01T10:12:16.6638770Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2022-12-01T10:12:16.6639068Z }
2022-12-01T10:12:16.6639327Z echo "${GITHUB_WORKSPACE}"
2022-12-01T10:12:16.6639611Z if [ -z "${NO_SUDO}" ]; then
2022-12-01T10:12:16.6639927Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2022-12-01T10:12:16.6640420Z else
2022-12-01T10:12:16.6640678Z   retry rm -rf "${GITHUB_WORKSPACE}"
2022-12-01T10:12:16.6640966Z fi
2022-12-01T10:12:16.6641299Z mkdir "${GITHUB_WORKSPACE}"
2022-12-01T10:12:16.6660458Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:12:16.6660775Z env:
2022-12-01T10:12:16.6661028Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:12:16.6661286Z   NO_SUDO: 
2022-12-01T10:12:16.6661506Z ##[endgroup]
2022-12-01T10:12:16.6897586Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:12:19.6152284Z ##[group]Run malfet/checkout@silent-checkout
2022-12-01T10:12:19.6152618Z with:
2022-12-01T10:12:19.6152902Z   ref: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:12:19.6153194Z   fetch-depth: 0
2022-12-01T10:12:19.6153431Z   submodules: recursive
2022-12-01T10:12:19.6153691Z   quiet-checkout: true
2022-12-01T10:12:19.6153969Z   repository: pytorch/pytorch
2022-12-01T10:12:19.6154423Z   token: ***
2022-12-01T10:12:19.6154686Z   ssh-strict: true
2022-12-01T10:12:19.6154954Z   persist-credentials: true
2022-12-01T10:12:19.6155221Z   clean: true
2022-12-01T10:12:19.6155445Z   lfs: false
2022-12-01T10:12:19.6155697Z   set-safe-directory: true
2022-12-01T10:12:19.6155944Z env:
2022-12-01T10:12:19.6156165Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:12:19.6156421Z ##[endgroup]
2022-12-01T10:12:19.7757498Z Syncing repository: pytorch/pytorch
2022-12-01T10:12:19.7759325Z ##[group]Getting Git version info
2022-12-01T10:12:19.7759866Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-12-01T10:12:19.7760450Z [command]/usr/bin/git version
2022-12-01T10:12:19.7760730Z git version 2.37.1
2022-12-01T10:12:19.7770756Z ##[endgroup]
2022-12-01T10:12:19.7790534Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7d223ccd-7abd-4ace-9f13-ae741bf62683' before making global git config changes
2022-12-01T10:12:19.7791977Z Adding repository directory to the temporary git global config as a safe directory
2022-12-01T10:12:19.7798127Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:12:19.7843458Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2022-12-01T10:12:19.7849994Z ##[group]Initializing the repository
2022-12-01T10:12:19.7853648Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:12:19.7887062Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-12-01T10:12:19.7887505Z hint: is subject to change. To configure the initial branch name to use in all
2022-12-01T10:12:19.7887941Z hint: of your new repositories, which will suppress this warning, call:
2022-12-01T10:12:19.7888260Z hint: 
2022-12-01T10:12:19.7888623Z hint: 	git config --global init.defaultBranch <name>
2022-12-01T10:12:19.7888918Z hint: 
2022-12-01T10:12:19.7889313Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-12-01T10:12:19.7889819Z hint: 'development'. The just-created branch can be renamed via this command:
2022-12-01T10:12:19.7890120Z hint: 
2022-12-01T10:12:19.7890568Z hint: 	git branch -m <name>
2022-12-01T10:12:19.7891145Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2022-12-01T10:12:19.7901461Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2022-12-01T10:12:19.7937235Z ##[endgroup]
2022-12-01T10:12:19.7937763Z ##[group]Disabling automatic garbage collection
2022-12-01T10:12:19.7941719Z [command]/usr/bin/git config --local gc.auto 0
2022-12-01T10:12:19.7971922Z ##[endgroup]
2022-12-01T10:12:19.7972775Z ##[group]Setting up auth
2022-12-01T10:12:19.7981950Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-12-01T10:12:19.8016088Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-12-01T10:12:19.8317788Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-12-01T10:12:19.8347650Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-12-01T10:12:19.8647180Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-12-01T10:12:19.8695351Z ##[endgroup]
2022-12-01T10:12:19.8695850Z ##[group]Fetching the repository
2022-12-01T10:12:19.8704151Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2022-12-01T10:13:15.0109915Z [command]/usr/bin/git rev-parse --verify --quiet c13d400bffe90e16b96520bbc8a41a6f0c9cd584^{object}
2022-12-01T10:13:15.0152571Z [command]/usr/bin/git -c protocol.version=2 fetch --no-tags --prune --quiet --no-recurse-submodules origin c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:13:16.4418604Z ##[endgroup]
2022-12-01T10:13:16.4419291Z ##[group]Determining the checkout info
2022-12-01T10:13:16.4420967Z ##[endgroup]
2022-12-01T10:13:16.4421392Z ##[group]Checking out the ref
2022-12-01T10:13:16.4426347Z [command]/usr/bin/git checkout --quiet --force c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:13:18.1267752Z ##[endgroup]
2022-12-01T10:13:18.1268329Z ##[group]Setting up auth for fetching submodules
2022-12-01T10:13:18.1274488Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2022-12-01T10:13:18.1331389Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2022-12-01T10:13:18.1367282Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2022-12-01T10:13:18.1402351Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2022-12-01T10:13:18.1432828Z ##[endgroup]
2022-12-01T10:13:18.1433293Z ##[group]Fetching submodules
2022-12-01T10:13:18.1438151Z [command]/usr/bin/git submodule sync --recursive
2022-12-01T10:13:18.1764690Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2022-12-01T10:13:18.2072721Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2022-12-01T10:13:18.2075391Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2022-12-01T10:13:18.2078460Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2022-12-01T10:13:18.2081750Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2022-12-01T10:13:18.2085268Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2022-12-01T10:13:18.2089193Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2022-12-01T10:13:18.2092806Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2022-12-01T10:13:18.2096661Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2022-12-01T10:13:18.2100665Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2022-12-01T10:13:18.2104805Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2022-12-01T10:13:18.2109277Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2022-12-01T10:13:18.2114833Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2022-12-01T10:13:18.2119257Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2022-12-01T10:13:18.2123829Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2022-12-01T10:13:18.2129142Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2022-12-01T10:13:18.2134194Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2022-12-01T10:13:18.2139592Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2022-12-01T10:13:18.2144782Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:13:18.2150483Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2022-12-01T10:13:18.2156145Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2022-12-01T10:13:18.2161727Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2022-12-01T10:13:18.2167666Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2022-12-01T10:13:18.2173447Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2022-12-01T10:13:18.2179414Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2022-12-01T10:13:18.2185477Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2022-12-01T10:13:18.2192526Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2022-12-01T10:13:18.2198817Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2022-12-01T10:13:18.2205299Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2022-12-01T10:13:18.2211968Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2022-12-01T10:13:18.2218700Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2022-12-01T10:13:18.2225590Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2022-12-01T10:13:18.2233419Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2022-12-01T10:13:18.2240485Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2022-12-01T10:13:18.2247833Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2022-12-01T10:13:18.2255102Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2022-12-01T10:13:18.2262621Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2022-12-01T10:13:18.2270715Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2022-12-01T10:13:18.2279242Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2022-12-01T10:13:18.2287155Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2022-12-01T10:13:18.2295428Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2022-12-01T10:13:18.2303678Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2022-12-01T10:13:18.2332482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2022-12-01T10:13:18.5443854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2022-12-01T10:13:18.7961464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2022-12-01T10:13:19.0215364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2022-12-01T10:13:19.3226701Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2022-12-01T10:13:19.5974358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2022-12-01T10:13:21.6300981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2022-12-01T10:13:29.5457538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2022-12-01T10:13:30.0657109Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2022-12-01T10:13:30.6683415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2022-12-01T10:13:32.4097544Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2022-12-01T10:13:33.7692936Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2022-12-01T10:13:35.4164563Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2022-12-01T10:13:42.9841441Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2022-12-01T10:13:43.8294968Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2022-12-01T10:13:46.1106063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2022-12-01T10:13:47.2832631Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2022-12-01T10:13:47.5043458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2022-12-01T10:13:48.0167455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2022-12-01T10:13:49.4578919Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2022-12-01T10:13:50.5330273Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2022-12-01T10:13:50.9816777Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2022-12-01T10:13:51.1870566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2022-12-01T10:13:51.4564618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2022-12-01T10:13:53.1156242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2022-12-01T10:13:53.6922112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2022-12-01T10:13:54.1050754Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2022-12-01T10:14:00.6985272Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2022-12-01T10:14:02.3498719Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2022-12-01T10:14:03.0009896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2022-12-01T10:14:03.2874439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2022-12-01T10:14:09.5479005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2022-12-01T10:14:09.7792664Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2022-12-01T10:14:10.0174210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2022-12-01T10:14:10.9053065Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2022-12-01T10:14:11.1521021Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2022-12-01T10:14:11.5089789Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2022-12-01T10:14:11.8431079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2022-12-01T10:14:12.4371358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2022-12-01T10:14:15.0401691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2022-12-01T10:14:15.5692062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2022-12-01T10:14:17.8408160Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2022-12-01T10:14:17.8534367Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2022-12-01T10:14:17.8630455Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2022-12-01T10:14:17.8914802Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2022-12-01T10:14:17.9190085Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2022-12-01T10:14:17.9616854Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2022-12-01T10:14:18.7289296Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2022-12-01T10:14:18.7545799Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-12-01T10:14:18.8787089Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2022-12-01T10:14:18.9195829Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2022-12-01T10:14:19.2793880Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2022-12-01T10:14:19.8176653Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2022-12-01T10:14:20.1202501Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2022-12-01T10:14:20.1778876Z Submodule path 'third_party/fbgemm': checked out '0d98c261561524cce92e37fe307ea6596664309a'
2022-12-01T10:14:20.1796740Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:20.1800714Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:20.1804695Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:20.1809002Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:20.1836644Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2022-12-01T10:14:21.1680886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2022-12-01T10:14:21.7479605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2022-12-01T10:14:22.8419680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2022-12-01T10:14:23.2367974Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2022-12-01T10:14:23.3626954Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2022-12-01T10:14:23.4333185Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2022-12-01T10:14:23.4450111Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2022-12-01T10:14:23.5522293Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2022-12-01T10:14:23.5955878Z Submodule path 'third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2022-12-01T10:14:23.6059668Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2022-12-01T10:14:23.6545345Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2022-12-01T10:14:23.6843394Z Submodule path 'third_party/gloo': checked out '5b143513263133af2b95547e97c07cebeb72bf72'
2022-12-01T10:14:23.7411404Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2022-12-01T10:14:23.7548412Z Submodule path 'third_party/ideep': checked out '77d662b313a762e82b389d3fd965e0098f12cd99'
2022-12-01T10:14:23.7566008Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:23.7592931Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2022-12-01T10:14:32.0894655Z Submodule path 'third_party/ideep/mkl-dnn': checked out '888a87a954e4fddb4d81fd10858eb834f2441b46'
2022-12-01T10:14:32.0914303Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:32.0941667Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2022-12-01T10:14:40.4862687Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out '52b5f107dd9cf10910aaa19cb47f3abf9b349815'
2022-12-01T10:14:40.4983136Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2022-12-01T10:14:40.5156433Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2022-12-01T10:14:40.6302645Z Submodule path 'third_party/kineto': checked out '0703c78999061b8329dfab7ec5046fc5764a5573'
2022-12-01T10:14:40.6320643Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:40.6323764Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:40.6351900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2022-12-01T10:14:41.7683629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2022-12-01T10:14:42.8643746Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2022-12-01T10:14:42.9308763Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2022-12-01T10:14:42.9555699Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2022-12-01T10:14:42.9721410Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2022-12-01T10:14:43.1050714Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2022-12-01T10:14:43.4204864Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2022-12-01T10:14:43.4238143Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:43.4241314Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:43.4269010Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2022-12-01T10:14:43.9616547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2022-12-01T10:14:44.8756485Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2022-12-01T10:14:44.9133465Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2022-12-01T10:14:44.9321625Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2022-12-01T10:14:44.9340031Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:44.9367295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2022-12-01T10:14:46.7381994Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2022-12-01T10:14:46.7405957Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:46.7409010Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:46.7438918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2022-12-01T10:14:47.2346510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2022-12-01T10:14:48.0930102Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2022-12-01T10:14:48.1720096Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2022-12-01T10:14:48.1727737Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:48.1756072Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2022-12-01T10:14:48.4303224Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-12-01T10:14:48.4407903Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2022-12-01T10:14:48.7601641Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2022-12-01T10:14:48.7624733Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:48.7627631Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:48.7656654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2022-12-01T10:14:49.2420950Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2022-12-01T10:14:50.2700554Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2022-12-01T10:14:50.3531574Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2022-12-01T10:14:50.3626925Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2022-12-01T10:14:50.3757141Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2022-12-01T10:14:50.4153890Z Submodule path 'third_party/pybind11': checked out 'aa304c9c7d725ffb9d10af08a3b34cb372307020'
2022-12-01T10:14:50.4254122Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2022-12-01T10:14:50.4600116Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2022-12-01T10:14:50.4708127Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2022-12-01T10:14:50.5252142Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2022-12-01T10:14:50.6614566Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2022-12-01T10:14:50.6930015Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2022-12-01T10:14:50.6948099Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:50.6951279Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:50.6955350Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:50.6958726Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:50.6986930Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2022-12-01T10:14:52.7818011Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2022-12-01T10:14:53.1421865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2022-12-01T10:14:54.5524469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2022-12-01T10:14:55.4697408Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2022-12-01T10:14:55.4877818Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2022-12-01T10:14:55.5669749Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2022-12-01T10:14:55.5998847Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2022-12-01T10:14:55.6015099Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:55.6042325Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2022-12-01T10:14:55.9458160Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2022-12-01T10:14:56.1075004Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2022-12-01T10:14:56.1108462Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2022-12-01T10:14:56.1440752Z Entering 'android/libs/fbjni'
2022-12-01T10:14:56.1484569Z Entering 'third_party/FP16'
2022-12-01T10:14:56.1528770Z Entering 'third_party/FXdiv'
2022-12-01T10:14:56.1572826Z Entering 'third_party/NNPACK'
2022-12-01T10:14:56.1618055Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:56.1662710Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:56.1706937Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:56.1762951Z Entering 'third_party/benchmark'
2022-12-01T10:14:56.1806382Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:56.1850834Z Entering 'third_party/cub'
2022-12-01T10:14:56.1895571Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:56.1946795Z Entering 'third_party/cutlass'
2022-12-01T10:14:56.1998545Z Entering 'third_party/eigen'
2022-12-01T10:14:56.2045192Z Entering 'third_party/fbgemm'
2022-12-01T10:14:56.2088923Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:56.2132187Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:56.2176117Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:56.2219086Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:56.2262835Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:56.2308587Z Entering 'third_party/fmt'
2022-12-01T10:14:56.2352379Z Entering 'third_party/foxi'
2022-12-01T10:14:56.2396003Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:56.2440026Z Entering 'third_party/gloo'
2022-12-01T10:14:56.2483648Z Entering 'third_party/googletest'
2022-12-01T10:14:56.2527874Z Entering 'third_party/ideep'
2022-12-01T10:14:56.2570421Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:56.2615367Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:56.2666177Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:56.2710480Z Entering 'third_party/ittapi'
2022-12-01T10:14:56.2753941Z Entering 'third_party/kineto'
2022-12-01T10:14:56.2799828Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:56.2846108Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:56.2891295Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:56.2936967Z Entering 'third_party/neon2sse'
2022-12-01T10:14:56.2980432Z Entering 'third_party/nlohmann'
2022-12-01T10:14:56.3026136Z Entering 'third_party/onnx'
2022-12-01T10:14:56.3092609Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:56.3135366Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:56.3182226Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:56.3224728Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:56.3273159Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:56.3316408Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:56.3359547Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:56.3407690Z Entering 'third_party/pocketfft'
2022-12-01T10:14:56.3450907Z Entering 'third_party/protobuf'
2022-12-01T10:14:56.3499706Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:56.3542498Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:56.3587034Z Entering 'third_party/psimd'
2022-12-01T10:14:56.3629784Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:56.3673136Z Entering 'third_party/pybind11'
2022-12-01T10:14:56.3716704Z Entering 'third_party/python-enum'
2022-12-01T10:14:56.3759528Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:56.3802311Z Entering 'third_party/python-six'
2022-12-01T10:14:56.3844888Z Entering 'third_party/sleef'
2022-12-01T10:14:56.3888546Z Entering 'third_party/tbb'
2022-12-01T10:14:56.3933325Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:56.3977472Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:56.4019808Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:56.4063430Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:56.4104919Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:56.4146801Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:56.4192350Z Entering 'third_party/zstd'
2022-12-01T10:14:56.4247020Z ##[endgroup]
2022-12-01T10:14:56.4247983Z ##[group]Persisting credentials for submodules
2022-12-01T10:14:56.4254973Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2022-12-01T10:14:56.4571511Z Entering 'android/libs/fbjni'
2022-12-01T10:14:56.4614860Z Entering 'third_party/FP16'
2022-12-01T10:14:56.4657518Z Entering 'third_party/FXdiv'
2022-12-01T10:14:56.4699046Z Entering 'third_party/NNPACK'
2022-12-01T10:14:56.4742076Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:56.4784174Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:56.4825923Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:56.4880047Z Entering 'third_party/benchmark'
2022-12-01T10:14:56.4922713Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:56.4965210Z Entering 'third_party/cub'
2022-12-01T10:14:56.5007728Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:56.5054875Z Entering 'third_party/cutlass'
2022-12-01T10:14:56.5104129Z Entering 'third_party/eigen'
2022-12-01T10:14:56.5148125Z Entering 'third_party/fbgemm'
2022-12-01T10:14:56.5190244Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:56.5231012Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:56.5273299Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:56.5315033Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:56.5358399Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:56.5402878Z Entering 'third_party/fmt'
2022-12-01T10:14:56.5444307Z Entering 'third_party/foxi'
2022-12-01T10:14:56.5485667Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:56.5527547Z Entering 'third_party/gloo'
2022-12-01T10:14:56.5569798Z Entering 'third_party/googletest'
2022-12-01T10:14:56.5611876Z Entering 'third_party/ideep'
2022-12-01T10:14:56.5652738Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:56.5695579Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:56.5743506Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:56.5784985Z Entering 'third_party/ittapi'
2022-12-01T10:14:56.5826516Z Entering 'third_party/kineto'
2022-12-01T10:14:56.5867599Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:56.5909224Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:56.5952203Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:56.5994822Z Entering 'third_party/neon2sse'
2022-12-01T10:14:56.6036722Z Entering 'third_party/nlohmann'
2022-12-01T10:14:56.6080899Z Entering 'third_party/onnx'
2022-12-01T10:14:56.6135855Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:56.6179644Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:56.6224480Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:56.6265534Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:56.6313042Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:56.6355917Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:56.6397373Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:56.6443672Z Entering 'third_party/pocketfft'
2022-12-01T10:14:56.6485255Z Entering 'third_party/protobuf'
2022-12-01T10:14:56.6530420Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:56.6571996Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:56.6616738Z Entering 'third_party/psimd'
2022-12-01T10:14:56.6659108Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:56.6701213Z Entering 'third_party/pybind11'
2022-12-01T10:14:56.6743368Z Entering 'third_party/python-enum'
2022-12-01T10:14:56.6785708Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:56.6827566Z Entering 'third_party/python-six'
2022-12-01T10:14:56.6869117Z Entering 'third_party/sleef'
2022-12-01T10:14:56.6910971Z Entering 'third_party/tbb'
2022-12-01T10:14:56.6958188Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:56.7001231Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:56.7043311Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:56.7084796Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:56.7126404Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:56.7167071Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:56.7212755Z Entering 'third_party/zstd'
2022-12-01T10:14:56.7268391Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2022-12-01T10:14:56.7585032Z Entering 'android/libs/fbjni'
2022-12-01T10:14:56.7624071Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2022-12-01T10:14:56.7642640Z Entering 'third_party/FP16'
2022-12-01T10:14:56.7682049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2022-12-01T10:14:56.7699596Z Entering 'third_party/FXdiv'
2022-12-01T10:14:56.7739824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2022-12-01T10:14:56.7758172Z Entering 'third_party/NNPACK'
2022-12-01T10:14:56.7796913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2022-12-01T10:14:56.7814362Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:56.7853542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2022-12-01T10:14:56.7871939Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:56.7909978Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2022-12-01T10:14:56.7927800Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:56.7967008Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2022-12-01T10:14:56.7996047Z Entering 'third_party/benchmark'
2022-12-01T10:14:56.8034738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:56.8052045Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:56.8092058Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2022-12-01T10:14:56.8109972Z Entering 'third_party/cub'
2022-12-01T10:14:56.8148951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2022-12-01T10:14:56.8167228Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:56.8206255Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2022-12-01T10:14:56.8229471Z Entering 'third_party/cutlass'
2022-12-01T10:14:56.8270087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2022-12-01T10:14:56.8294266Z Entering 'third_party/eigen'
2022-12-01T10:14:56.8334115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2022-12-01T10:14:56.8354741Z Entering 'third_party/fbgemm'
2022-12-01T10:14:56.8394533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2022-12-01T10:14:56.8412349Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:56.8452801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2022-12-01T10:14:56.8470305Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:56.8509294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2022-12-01T10:14:56.8527587Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:56.8566895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:56.8584215Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:56.8623260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2022-12-01T10:14:56.8642539Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:56.8682502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2022-12-01T10:14:56.8702322Z Entering 'third_party/fmt'
2022-12-01T10:14:56.8741932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2022-12-01T10:14:56.8759906Z Entering 'third_party/foxi'
2022-12-01T10:14:56.8799202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2022-12-01T10:14:56.8816399Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:56.8855728Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2022-12-01T10:14:56.8873706Z Entering 'third_party/gloo'
2022-12-01T10:14:56.8912857Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2022-12-01T10:14:56.8930331Z Entering 'third_party/googletest'
2022-12-01T10:14:56.8969775Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:56.8987451Z Entering 'third_party/ideep'
2022-12-01T10:14:56.9026288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2022-12-01T10:14:56.9043863Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:56.9081927Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2022-12-01T10:14:56.9100794Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:56.9139778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2022-12-01T10:14:56.9163828Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:56.9202266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2022-12-01T10:14:56.9219568Z Entering 'third_party/ittapi'
2022-12-01T10:14:56.9258861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2022-12-01T10:14:56.9277547Z Entering 'third_party/kineto'
2022-12-01T10:14:56.9316849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2022-12-01T10:14:56.9333786Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:56.9372817Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2022-12-01T10:14:56.9390254Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:56.9428788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:56.9448494Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:56.9487996Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2022-12-01T10:14:56.9505560Z Entering 'third_party/neon2sse'
2022-12-01T10:14:56.9545338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2022-12-01T10:14:56.9562684Z Entering 'third_party/nlohmann'
2022-12-01T10:14:56.9602201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2022-12-01T10:14:56.9621143Z Entering 'third_party/onnx'
2022-12-01T10:14:56.9660233Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2022-12-01T10:14:56.9691061Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:56.9730294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:56.9748731Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:56.9788047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:56.9807994Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:56.9847679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2022-12-01T10:14:56.9864411Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:56.9903508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2022-12-01T10:14:56.9926082Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:56.9965465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:56.9983513Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:57.0023398Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:57.0040942Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:57.0081060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-12-01T10:14:57.0104372Z Entering 'third_party/pocketfft'
2022-12-01T10:14:57.0143903Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2022-12-01T10:14:57.0161722Z Entering 'third_party/protobuf'
2022-12-01T10:14:57.0201016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2022-12-01T10:14:57.0222080Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:57.0260893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2022-12-01T10:14:57.0278623Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:57.0317378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:57.0336794Z Entering 'third_party/psimd'
2022-12-01T10:14:57.0376330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2022-12-01T10:14:57.0394378Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:57.0433893Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2022-12-01T10:14:57.0450974Z Entering 'third_party/pybind11'
2022-12-01T10:14:57.0489788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:57.0507416Z Entering 'third_party/python-enum'
2022-12-01T10:14:57.0547570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2022-12-01T10:14:57.0564847Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:57.0603521Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2022-12-01T10:14:57.0620820Z Entering 'third_party/python-six'
2022-12-01T10:14:57.0660318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2022-12-01T10:14:57.0677847Z Entering 'third_party/sleef'
2022-12-01T10:14:57.0716539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2022-12-01T10:14:57.0734171Z Entering 'third_party/tbb'
2022-12-01T10:14:57.0773226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2022-12-01T10:14:57.0793007Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:57.0832570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2022-12-01T10:14:57.0849904Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:57.0888360Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2022-12-01T10:14:57.0905498Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:57.0944035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2022-12-01T10:14:57.0961830Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:57.1000294Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2022-12-01T10:14:57.1018078Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:57.1057672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2022-12-01T10:14:57.1074864Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:57.1114717Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2022-12-01T10:14:57.1135268Z Entering 'third_party/zstd'
2022-12-01T10:14:57.1174626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2022-12-01T10:14:57.2334380Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2022-12-01T10:14:57.2651708Z Entering 'android/libs/fbjni'
2022-12-01T10:14:57.2694937Z Entering 'third_party/FP16'
2022-12-01T10:14:57.2737764Z Entering 'third_party/FXdiv'
2022-12-01T10:14:57.2780569Z Entering 'third_party/NNPACK'
2022-12-01T10:14:57.2824674Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:57.2868082Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:57.2911570Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:57.2966642Z Entering 'third_party/benchmark'
2022-12-01T10:14:57.3010382Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:57.3053583Z Entering 'third_party/cub'
2022-12-01T10:14:57.3097290Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:57.3145683Z Entering 'third_party/cutlass'
2022-12-01T10:14:57.3196838Z Entering 'third_party/eigen'
2022-12-01T10:14:57.3241529Z Entering 'third_party/fbgemm'
2022-12-01T10:14:57.3284028Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:57.3327450Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:57.3371069Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:57.3414419Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:57.3457811Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:57.3503097Z Entering 'third_party/fmt'
2022-12-01T10:14:57.3546916Z Entering 'third_party/foxi'
2022-12-01T10:14:57.3591466Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:57.3634912Z Entering 'third_party/gloo'
2022-12-01T10:14:57.3678489Z Entering 'third_party/googletest'
2022-12-01T10:14:57.3722447Z Entering 'third_party/ideep'
2022-12-01T10:14:57.3763953Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:57.3808837Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:57.3859457Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:57.3902914Z Entering 'third_party/ittapi'
2022-12-01T10:14:57.3947874Z Entering 'third_party/kineto'
2022-12-01T10:14:57.3990540Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:57.4033603Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:57.4078713Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:57.4124095Z Entering 'third_party/neon2sse'
2022-12-01T10:14:57.4166686Z Entering 'third_party/nlohmann'
2022-12-01T10:14:57.4211524Z Entering 'third_party/onnx'
2022-12-01T10:14:57.4267346Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:57.4311192Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:57.4356153Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:57.4398429Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:57.4447123Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:57.4490718Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:57.4533817Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:57.4581518Z Entering 'third_party/pocketfft'
2022-12-01T10:14:57.4624370Z Entering 'third_party/protobuf'
2022-12-01T10:14:57.4672122Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:57.4715124Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:57.4759975Z Entering 'third_party/psimd'
2022-12-01T10:14:57.4802015Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:57.4845321Z Entering 'third_party/pybind11'
2022-12-01T10:14:57.4887757Z Entering 'third_party/python-enum'
2022-12-01T10:14:57.4930530Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:57.4973117Z Entering 'third_party/python-six'
2022-12-01T10:14:57.5015624Z Entering 'third_party/sleef'
2022-12-01T10:14:57.5058281Z Entering 'third_party/tbb'
2022-12-01T10:14:57.5103197Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:57.5147311Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:57.5190724Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:57.5232853Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:57.5275944Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:57.5318185Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:57.5363757Z Entering 'third_party/zstd'
2022-12-01T10:14:57.5419478Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2022-12-01T10:14:57.5737199Z Entering 'android/libs/fbjni'
2022-12-01T10:14:57.5779868Z Entering 'third_party/FP16'
2022-12-01T10:14:57.5823724Z Entering 'third_party/FXdiv'
2022-12-01T10:14:57.5866838Z Entering 'third_party/NNPACK'
2022-12-01T10:14:57.5909738Z Entering 'third_party/QNNPACK'
2022-12-01T10:14:57.5953827Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T10:14:57.5997603Z Entering 'third_party/XNNPACK'
2022-12-01T10:14:57.6052084Z Entering 'third_party/benchmark'
2022-12-01T10:14:57.6095129Z Entering 'third_party/cpuinfo'
2022-12-01T10:14:57.6139108Z Entering 'third_party/cub'
2022-12-01T10:14:57.6182315Z Entering 'third_party/cudnn_frontend'
2022-12-01T10:14:57.6232158Z Entering 'third_party/cutlass'
2022-12-01T10:14:57.6283523Z Entering 'third_party/eigen'
2022-12-01T10:14:57.6328692Z Entering 'third_party/fbgemm'
2022-12-01T10:14:57.6372190Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T10:14:57.6414500Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T10:14:57.6458715Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T10:14:57.6500844Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T10:14:57.6544193Z Entering 'third_party/flatbuffers'
2022-12-01T10:14:57.6588945Z Entering 'third_party/fmt'
2022-12-01T10:14:57.6632707Z Entering 'third_party/foxi'
2022-12-01T10:14:57.6675436Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T10:14:57.6718909Z Entering 'third_party/gloo'
2022-12-01T10:14:57.6761673Z Entering 'third_party/googletest'
2022-12-01T10:14:57.6804758Z Entering 'third_party/ideep'
2022-12-01T10:14:57.6847167Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T10:14:57.6891241Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T10:14:57.6941324Z Entering 'third_party/ios-cmake'
2022-12-01T10:14:57.6984837Z Entering 'third_party/ittapi'
2022-12-01T10:14:57.7027554Z Entering 'third_party/kineto'
2022-12-01T10:14:57.7069746Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T10:14:57.7112759Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T10:14:57.7156514Z Entering 'third_party/nccl/nccl'
2022-12-01T10:14:57.7201944Z Entering 'third_party/neon2sse'
2022-12-01T10:14:57.7243818Z Entering 'third_party/nlohmann'
2022-12-01T10:14:57.7287572Z Entering 'third_party/onnx'
2022-12-01T10:14:57.7344003Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T10:14:57.7386818Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T10:14:57.7431646Z Entering 'third_party/onnx-tensorrt'
2022-12-01T10:14:57.7473462Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T10:14:57.7521287Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T10:14:57.7564312Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T10:14:57.7607351Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T10:14:57.7654624Z Entering 'third_party/pocketfft'
2022-12-01T10:14:57.7697864Z Entering 'third_party/protobuf'
2022-12-01T10:14:57.7743684Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T10:14:57.7786556Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T10:14:57.7830576Z Entering 'third_party/psimd'
2022-12-01T10:14:57.7873821Z Entering 'third_party/pthreadpool'
2022-12-01T10:14:57.7916979Z Entering 'third_party/pybind11'
2022-12-01T10:14:57.7959984Z Entering 'third_party/python-enum'
2022-12-01T10:14:57.8003858Z Entering 'third_party/python-peachpy'
2022-12-01T10:14:57.8046215Z Entering 'third_party/python-six'
2022-12-01T10:14:57.8089056Z Entering 'third_party/sleef'
2022-12-01T10:14:57.8131741Z Entering 'third_party/tbb'
2022-12-01T10:14:57.8177520Z Entering 'third_party/tensorpipe'
2022-12-01T10:14:57.8221361Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T10:14:57.8264026Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T10:14:57.8307020Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T10:14:57.8350041Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T10:14:57.8392199Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T10:14:57.8437540Z Entering 'third_party/zstd'
2022-12-01T10:14:57.8490909Z ##[endgroup]
2022-12-01T10:14:57.8534490Z [command]/usr/bin/git log -1 --format='%H'
2022-12-01T10:14:57.8566419Z 'c13d400bffe90e16b96520bbc8a41a6f0c9cd584'
2022-12-01T10:14:57.8739756Z Prepare all required actions
2022-12-01T10:14:57.8773872Z ##[group]Run ./.github/actions/setup-linux
2022-12-01T10:14:57.8774153Z env:
2022-12-01T10:14:57.8774398Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:57.8774638Z ##[endgroup]
2022-12-01T10:14:57.8834969Z ##[group]Run set -euo pipefail
2022-12-01T10:14:57.8835283Z set -euo pipefail
2022-12-01T10:14:57.8835570Z function get_ec2_metadata() {
2022-12-01T10:14:57.8835904Z   # Pulled from instance metadata endpoint for EC2
2022-12-01T10:14:57.8836378Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2022-12-01T10:14:57.8836764Z   category=$1
2022-12-01T10:14:57.8837092Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2022-12-01T10:14:57.8837549Z }
2022-12-01T10:14:57.8837812Z echo "ami-id: $(get_ec2_metadata ami-id)"
2022-12-01T10:14:57.8838197Z echo "instance-id: $(get_ec2_metadata instance-id)"
2022-12-01T10:14:57.8838575Z echo "instance-type: $(get_ec2_metadata instance-type)"
2022-12-01T10:14:57.8838900Z echo "system info $(uname -a)"
2022-12-01T10:14:57.8852711Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:57.8853009Z env:
2022-12-01T10:14:57.8853232Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:57.8853488Z ##[endgroup]
2022-12-01T10:14:57.8954834Z ami-id: ami-096198a0bccc6bad4
2022-12-01T10:14:57.9016703Z instance-id: i-0d2a7b5e684093f31
2022-12-01T10:14:57.9077953Z instance-type: g3.8xlarge
2022-12-01T10:14:57.9086502Z system info Linux ip-10-0-3-202.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2022-12-01T10:14:57.9106638Z ##[group]Run if systemctl is-active --quiet docker; then
2022-12-01T10:14:57.9107013Z if systemctl is-active --quiet docker; then
2022-12-01T10:14:57.9107340Z     echo "Docker daemon is running...";
2022-12-01T10:14:57.9107602Z else
2022-12-01T10:14:57.9107909Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2022-12-01T10:14:57.9108214Z fi
2022-12-01T10:14:57.9121166Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:57.9121445Z env:
2022-12-01T10:14:57.9121684Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:57.9121938Z ##[endgroup]
2022-12-01T10:14:57.9174204Z Docker daemon is running...
2022-12-01T10:14:57.9195484Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-12-01T10:14:57.9195957Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2022-12-01T10:14:57.9196373Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:14:57.9196859Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2022-12-01T10:14:57.9197336Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2022-12-01T10:14:57.9209545Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:57.9209839Z env:
2022-12-01T10:14:57.9210059Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:57.9210326Z   AWS_RETRY_MODE: standard
2022-12-01T10:14:57.9210583Z   AWS_MAX_ATTEMPTS: 5
2022-12-01T10:14:57.9210835Z   AWS_DEFAULT_REGION: us-east-1
2022-12-01T10:14:57.9211095Z ##[endgroup]
2022-12-01T10:14:58.8712219Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2022-12-01T10:14:58.8712945Z Configure a credential helper to remove this warning. See
2022-12-01T10:14:58.8713925Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2022-12-01T10:14:58.8714398Z 
2022-12-01T10:14:58.8714550Z Login Succeeded
2022-12-01T10:14:58.8766302Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:14:58.8767101Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:14:58.8768025Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2022-12-01T10:14:58.8789045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:58.8790073Z env:
2022-12-01T10:14:58.8790547Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:58.8791026Z ##[endgroup]
2022-12-01T10:14:58.8891598Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2022-12-01T10:14:58.8891943Z with:
2022-12-01T10:14:58.8892381Z   github-secret: ***
2022-12-01T10:14:58.8892661Z   activate-with-label: false
2022-12-01T10:14:58.8892910Z   label: with-ssh
2022-12-01T10:14:58.8893170Z   remove-existing-keys: true
2022-12-01T10:14:58.8893420Z env:
2022-12-01T10:14:58.8893637Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:58.8893887Z ##[endgroup]
2022-12-01T10:14:59.3377259Z Grabbing public ssh keys from https://github.com/charlie-wt.keys
2022-12-01T10:14:59.4239328Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2022-12-01T10:14:59.4260974Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2022-12-01T10:14:59.4303239Z Login using: ssh ec2-user@ec2-34-233-124-119.compute-1.amazonaws.com
2022-12-01T10:14:59.4376277Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2022-12-01T10:14:59.4376641Z with:
2022-12-01T10:14:59.4377105Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:14:59.4377560Z env:
2022-12-01T10:14:59.4377797Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:59.4378036Z ##[endgroup]
2022-12-01T10:14:59.4396615Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:14:59.4397217Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2022-12-01T10:14:59.4397602Z # ignore output since only exit code is used for conditional
2022-12-01T10:14:59.4397992Z # only pull docker image if it's not available locally
2022-12-01T10:14:59.4398382Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2022-12-01T10:14:59.4398797Z   retry docker pull "${DOCKER_IMAGE}"
2022-12-01T10:14:59.4399075Z fi
2022-12-01T10:14:59.4412302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:14:59.4412599Z env:
2022-12-01T10:14:59.4412839Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:14:59.4413331Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:14:59.4413805Z ##[endgroup]
2022-12-01T10:14:59.6650815Z fa72f5a0a230eb632055220542038bd4ceca184b: Pulling from pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7
2022-12-01T10:14:59.6651281Z 40dd5be53814: Pulling fs layer
2022-12-01T10:14:59.6651582Z bd44602516a4: Pulling fs layer
2022-12-01T10:14:59.6651842Z 8ebfb31ea67d: Pulling fs layer
2022-12-01T10:14:59.6655572Z 1589dc294916: Pulling fs layer
2022-12-01T10:14:59.6656156Z 2c3a764ff1ef: Pulling fs layer
2022-12-01T10:14:59.6656698Z 2fb24fb5f7cb: Pulling fs layer
2022-12-01T10:14:59.6657204Z d6e4b45751c9: Pulling fs layer
2022-12-01T10:14:59.6657730Z 98a26bc0781e: Pulling fs layer
2022-12-01T10:14:59.6658063Z 07c42b0591b2: Pulling fs layer
2022-12-01T10:14:59.6658848Z 1589dc294916: Waiting
2022-12-01T10:14:59.6659311Z a7b4d9098b01: Pulling fs layer
2022-12-01T10:14:59.6659632Z 9b965084d7f3: Pulling fs layer
2022-12-01T10:14:59.6659890Z d6e4b45751c9: Waiting
2022-12-01T10:14:59.6660130Z 306938abc720: Pulling fs layer
2022-12-01T10:14:59.6660523Z 06e616940749: Pulling fs layer
2022-12-01T10:14:59.6660986Z 98a26bc0781e: Waiting
2022-12-01T10:14:59.6661273Z 9b965084d7f3: Waiting
2022-12-01T10:14:59.6661673Z 540637333efa: Pulling fs layer
2022-12-01T10:14:59.6662160Z 306938abc720: Waiting
2022-12-01T10:14:59.6662680Z 5cd3ebaf3e1b: Pulling fs layer
2022-12-01T10:14:59.6663026Z 06e616940749: Waiting
2022-12-01T10:14:59.6663287Z dd7a31bfac90: Pulling fs layer
2022-12-01T10:14:59.6663544Z 6416ca3f405b: Pulling fs layer
2022-12-01T10:14:59.6663814Z 19c44dd1f4cf: Pulling fs layer
2022-12-01T10:14:59.6664083Z 6416ca3f405b: Waiting
2022-12-01T10:14:59.6664319Z 33688bb73038: Pulling fs layer
2022-12-01T10:14:59.6664584Z 570b8809d75f: Pulling fs layer
2022-12-01T10:14:59.6664850Z 3ea8b8f31abf: Pulling fs layer
2022-12-01T10:14:59.6668332Z 19c44dd1f4cf: Waiting
2022-12-01T10:14:59.6668689Z 690eecb8c64c: Pulling fs layer
2022-12-01T10:14:59.6668966Z 570b8809d75f: Waiting
2022-12-01T10:14:59.6669230Z e2285545a8bf: Pulling fs layer
2022-12-01T10:14:59.6669833Z 555faed4e286: Pulling fs layer
2022-12-01T10:14:59.6670284Z c53c37f056ac: Pulling fs layer
2022-12-01T10:14:59.6670770Z 33688bb73038: Waiting
2022-12-01T10:14:59.6671113Z dd7a31bfac90: Waiting
2022-12-01T10:14:59.6671572Z e2285545a8bf: Waiting
2022-12-01T10:14:59.6672012Z 690eecb8c64c: Waiting
2022-12-01T10:14:59.6672519Z 75bd8ea35c95: Pulling fs layer
2022-12-01T10:14:59.6673093Z 2fb24fb5f7cb: Waiting
2022-12-01T10:14:59.6673585Z 555faed4e286: Waiting
2022-12-01T10:14:59.6673912Z b18c1d97e9da: Pulling fs layer
2022-12-01T10:14:59.6674190Z d90720873b3b: Pulling fs layer
2022-12-01T10:14:59.6674579Z 53b1597ec1c2: Pulling fs layer
2022-12-01T10:14:59.6674838Z 3ea8b8f31abf: Waiting
2022-12-01T10:14:59.6675094Z ef984347a312: Pulling fs layer
2022-12-01T10:14:59.6681071Z 343a1b6dc08c: Pulling fs layer
2022-12-01T10:14:59.6681773Z b18c1d97e9da: Waiting
2022-12-01T10:14:59.6682260Z 16fdefc9f32c: Pulling fs layer
2022-12-01T10:14:59.6682783Z de0913f5235c: Pulling fs layer
2022-12-01T10:14:59.6683253Z bf99e81d1939: Pulling fs layer
2022-12-01T10:14:59.6683720Z c0f36b953c44: Pulling fs layer
2022-12-01T10:14:59.6684196Z 7456bfaa70b2: Pulling fs layer
2022-12-01T10:14:59.6684848Z ef984347a312: Waiting
2022-12-01T10:14:59.6685336Z 68e8ef95b145: Pulling fs layer
2022-12-01T10:14:59.6685789Z e497d45bb922: Pulling fs layer
2022-12-01T10:14:59.6686234Z bf4a66dadd84: Pulling fs layer
2022-12-01T10:14:59.6686726Z 343a1b6dc08c: Waiting
2022-12-01T10:14:59.6687152Z bf99e81d1939: Waiting
2022-12-01T10:14:59.6687527Z 16fdefc9f32c: Waiting
2022-12-01T10:14:59.6687928Z 53b1597ec1c2: Waiting
2022-12-01T10:14:59.6688311Z 7456bfaa70b2: Waiting
2022-12-01T10:14:59.6688719Z fd0d4928961f: Pulling fs layer
2022-12-01T10:14:59.6689150Z 6df8346812f8: Pulling fs layer
2022-12-01T10:14:59.6689551Z e497d45bb922: Waiting
2022-12-01T10:14:59.6689938Z 8b46f8aa8681: Pulling fs layer
2022-12-01T10:14:59.6690467Z 6db5688b5394: Pulling fs layer
2022-12-01T10:14:59.6690856Z fd0d4928961f: Waiting
2022-12-01T10:14:59.6691249Z bf4a66dadd84: Waiting
2022-12-01T10:14:59.6691616Z daa053ac35f3: Pulling fs layer
2022-12-01T10:14:59.6692005Z 6db5688b5394: Waiting
2022-12-01T10:14:59.6692361Z c0f36b953c44: Waiting
2022-12-01T10:14:59.6692720Z c9a8257f8dc0: Pulling fs layer
2022-12-01T10:14:59.6693129Z 30f21817172d: Pulling fs layer
2022-12-01T10:14:59.6693509Z daa053ac35f3: Waiting
2022-12-01T10:14:59.6693879Z bb9282afce05: Pulling fs layer
2022-12-01T10:14:59.6694273Z 909c475d21ec: Pulling fs layer
2022-12-01T10:14:59.6694714Z 192b8944f15a: Pulling fs layer
2022-12-01T10:14:59.6695136Z 3dd92c832839: Pulling fs layer
2022-12-01T10:14:59.6695548Z bb9282afce05: Waiting
2022-12-01T10:14:59.6695919Z de0913f5235c: Waiting
2022-12-01T10:14:59.6696280Z 5e1b0146a21d: Pulling fs layer
2022-12-01T10:14:59.6696709Z 909c475d21ec: Waiting
2022-12-01T10:14:59.6697183Z c76cf0339419: Pulling fs layer
2022-12-01T10:14:59.6697653Z 2dc4928eecf0: Pulling fs layer
2022-12-01T10:14:59.6698288Z 68e8ef95b145: Waiting
2022-12-01T10:14:59.6698902Z 2dc4928eecf0: Waiting
2022-12-01T10:14:59.6699317Z 6df8346812f8: Waiting
2022-12-01T10:14:59.6699970Z 192b8944f15a: Waiting
2022-12-01T10:14:59.6700435Z 30f21817172d: Waiting
2022-12-01T10:14:59.8000852Z bd44602516a4: Verifying Checksum
2022-12-01T10:14:59.8001188Z bd44602516a4: Download complete
2022-12-01T10:14:59.8958244Z 1589dc294916: Verifying Checksum
2022-12-01T10:14:59.8958570Z 1589dc294916: Download complete
2022-12-01T10:14:59.9689529Z 2c3a764ff1ef: Verifying Checksum
2022-12-01T10:14:59.9689847Z 2c3a764ff1ef: Download complete
2022-12-01T10:14:59.9854296Z 8ebfb31ea67d: Verifying Checksum
2022-12-01T10:14:59.9854594Z 8ebfb31ea67d: Download complete
2022-12-01T10:14:59.9972491Z 40dd5be53814: Verifying Checksum
2022-12-01T10:14:59.9972790Z 40dd5be53814: Download complete
2022-12-01T10:15:00.0602546Z d6e4b45751c9: Verifying Checksum
2022-12-01T10:15:00.0602885Z d6e4b45751c9: Download complete
2022-12-01T10:15:00.1612507Z 07c42b0591b2: Download complete
2022-12-01T10:15:00.3123372Z a7b4d9098b01: Download complete
2022-12-01T10:15:00.7543415Z 40dd5be53814: Pull complete
2022-12-01T10:15:01.0492049Z bd44602516a4: Pull complete
2022-12-01T10:15:01.5785156Z 8ebfb31ea67d: Pull complete
2022-12-01T10:15:01.7087616Z 1589dc294916: Pull complete
2022-12-01T10:15:02.0579966Z 2c3a764ff1ef: Pull complete
2022-12-01T10:15:02.6541222Z 9b965084d7f3: Verifying Checksum
2022-12-01T10:15:02.6541613Z 9b965084d7f3: Download complete
2022-12-01T10:15:02.7378241Z 306938abc720: Verifying Checksum
2022-12-01T10:15:02.7378599Z 306938abc720: Download complete
2022-12-01T10:15:02.8352724Z 06e616940749: Verifying Checksum
2022-12-01T10:15:02.8353264Z 06e616940749: Download complete
2022-12-01T10:15:02.9552924Z 540637333efa: Download complete
2022-12-01T10:15:04.1557913Z 5cd3ebaf3e1b: Verifying Checksum
2022-12-01T10:15:04.1558263Z 5cd3ebaf3e1b: Download complete
2022-12-01T10:15:04.2516685Z dd7a31bfac90: Download complete
2022-12-01T10:15:04.3578216Z 6416ca3f405b: Verifying Checksum
2022-12-01T10:15:24.2745763Z 2fb24fb5f7cb: Verifying Checksum
2022-12-01T10:15:24.2746311Z 2fb24fb5f7cb: Download complete
2022-12-01T10:15:24.3721316Z 33688bb73038: Download complete
2022-12-01T10:15:24.4586581Z 570b8809d75f: Verifying Checksum
2022-12-01T10:15:24.4587339Z 570b8809d75f: Download complete
2022-12-01T10:15:24.5642460Z 3ea8b8f31abf: Verifying Checksum
2022-12-01T10:15:24.5643188Z 3ea8b8f31abf: Download complete
2022-12-01T10:15:24.6523061Z 690eecb8c64c: Verifying Checksum
2022-12-01T10:15:24.6523541Z 690eecb8c64c: Download complete
2022-12-01T10:15:24.8200716Z e2285545a8bf: Verifying Checksum
2022-12-01T10:15:24.8201393Z e2285545a8bf: Download complete
2022-12-01T10:15:24.9303290Z 555faed4e286: Verifying Checksum
2022-12-01T10:15:24.9304685Z 555faed4e286: Download complete
2022-12-01T10:15:27.4377919Z c53c37f056ac: Verifying Checksum
2022-12-01T10:15:27.4378306Z c53c37f056ac: Download complete
2022-12-01T10:15:27.5485499Z 75bd8ea35c95: Download complete
2022-12-01T10:15:27.6289324Z b18c1d97e9da: Verifying Checksum
2022-12-01T10:15:27.6289941Z b18c1d97e9da: Download complete
2022-12-01T10:15:27.7174463Z d90720873b3b: Download complete
2022-12-01T10:15:27.8000129Z 53b1597ec1c2: Verifying Checksum
2022-12-01T10:15:27.8000446Z 53b1597ec1c2: Download complete
2022-12-01T10:15:27.8849375Z ef984347a312: Verifying Checksum
2022-12-01T10:15:27.8849744Z ef984347a312: Download complete
2022-12-01T10:15:33.8781248Z 343a1b6dc08c: Verifying Checksum
2022-12-01T10:15:33.8781614Z 343a1b6dc08c: Download complete
2022-12-01T10:15:33.9270690Z 98a26bc0781e: Verifying Checksum
2022-12-01T10:15:33.9271021Z 98a26bc0781e: Download complete
2022-12-01T10:15:33.9487145Z 16fdefc9f32c: Verifying Checksum
2022-12-01T10:15:33.9488006Z 16fdefc9f32c: Download complete
2022-12-01T10:15:34.0124923Z de0913f5235c: Verifying Checksum
2022-12-01T10:15:34.0125544Z de0913f5235c: Download complete
2022-12-01T10:15:34.0985781Z c0f36b953c44: Download complete
2022-12-01T10:15:34.1815546Z 7456bfaa70b2: Verifying Checksum
2022-12-01T10:15:34.1815986Z 7456bfaa70b2: Download complete
2022-12-01T10:15:34.6694172Z bf99e81d1939: Verifying Checksum
2022-12-01T10:15:34.6694771Z bf99e81d1939: Download complete
2022-12-01T10:15:34.7224191Z 68e8ef95b145: Verifying Checksum
2022-12-01T10:15:34.7225164Z 68e8ef95b145: Download complete
2022-12-01T10:15:34.7487355Z e497d45bb922: Verifying Checksum
2022-12-01T10:15:34.7487655Z e497d45bb922: Download complete
2022-12-01T10:15:34.8335464Z fd0d4928961f: Verifying Checksum
2022-12-01T10:15:34.8335785Z fd0d4928961f: Download complete
2022-12-01T10:15:34.9030137Z 6df8346812f8: Verifying Checksum
2022-12-01T10:15:34.9030566Z 6df8346812f8: Download complete
2022-12-01T10:15:35.9429984Z bf4a66dadd84: Verifying Checksum
2022-12-01T10:15:35.9431077Z bf4a66dadd84: Download complete
2022-12-01T10:15:36.0307823Z 6db5688b5394: Download complete
2022-12-01T10:15:36.1186730Z daa053ac35f3: Download complete
2022-12-01T10:15:36.2169433Z c9a8257f8dc0: Verifying Checksum
2022-12-01T10:15:36.2169807Z c9a8257f8dc0: Download complete
2022-12-01T10:15:36.2980512Z 30f21817172d: Verifying Checksum
2022-12-01T10:15:36.2981067Z 30f21817172d: Download complete
2022-12-01T10:15:36.8252430Z bb9282afce05: Verifying Checksum
2022-12-01T10:15:36.8252795Z bb9282afce05: Download complete
2022-12-01T10:15:36.8945229Z 909c475d21ec: Download complete
2022-12-01T10:15:38.3643283Z 192b8944f15a: Verifying Checksum
2022-12-01T10:15:38.3643643Z 192b8944f15a: Download complete
2022-12-01T10:15:38.4298271Z 3dd92c832839: Verifying Checksum
2022-12-01T10:15:38.4298706Z 3dd92c832839: Download complete
2022-12-01T10:15:40.1334242Z 2fb24fb5f7cb: Pull complete
2022-12-01T10:15:42.0117349Z d6e4b45751c9: Pull complete
2022-12-01T10:15:45.0164218Z 8b46f8aa8681: Verifying Checksum
2022-12-01T10:15:45.0164592Z 8b46f8aa8681: Download complete
2022-12-01T10:15:45.1056851Z c76cf0339419: Download complete
2022-12-01T10:15:45.2002295Z 2dc4928eecf0: Verifying Checksum
2022-12-01T10:15:57.0237563Z 19c44dd1f4cf: Download complete
2022-12-01T10:16:06.1525720Z 98a26bc0781e: Pull complete
2022-12-01T10:16:08.0618380Z 07c42b0591b2: Pull complete
2022-12-01T10:16:09.9679459Z a7b4d9098b01: Pull complete
2022-12-01T10:16:18.3412422Z 9b965084d7f3: Pull complete
2022-12-01T10:16:20.2014446Z 306938abc720: Pull complete
2022-12-01T10:16:22.4319143Z 06e616940749: Pull complete
2022-12-01T10:16:25.2862127Z 540637333efa: Pull complete
2022-12-01T10:16:27.5173409Z 5cd3ebaf3e1b: Pull complete
2022-12-01T10:16:27.6454442Z dd7a31bfac90: Pull complete
2022-12-01T10:16:27.7687361Z 6416ca3f405b: Pull complete
2022-12-01T10:17:02.1485193Z 19c44dd1f4cf: Pull complete
2022-12-01T10:17:05.1926313Z 33688bb73038: Pull complete
2022-12-01T10:17:07.0198982Z 5e1b0146a21d: Verifying Checksum
2022-12-01T10:17:07.0199375Z 5e1b0146a21d: Download complete
2022-12-01T10:17:07.0636761Z 570b8809d75f: Pull complete
2022-12-01T10:17:09.5838353Z 3ea8b8f31abf: Pull complete
2022-12-01T10:17:11.4349595Z 690eecb8c64c: Pull complete
2022-12-01T10:17:13.5602404Z e2285545a8bf: Pull complete
2022-12-01T10:17:15.4668465Z 555faed4e286: Pull complete
2022-12-01T10:17:20.2241025Z c53c37f056ac: Pull complete
2022-12-01T10:17:21.0746904Z 75bd8ea35c95: Pull complete
2022-12-01T10:17:21.1798395Z b18c1d97e9da: Pull complete
2022-12-01T10:17:21.3271430Z d90720873b3b: Pull complete
2022-12-01T10:17:21.4274906Z 53b1597ec1c2: Pull complete
2022-12-01T10:17:21.5502438Z ef984347a312: Pull complete
2022-12-01T10:17:29.4270479Z 343a1b6dc08c: Pull complete
2022-12-01T10:17:32.3511588Z 16fdefc9f32c: Pull complete
2022-12-01T10:17:35.6691878Z de0913f5235c: Pull complete
2022-12-01T10:17:39.1366763Z bf99e81d1939: Pull complete
2022-12-01T10:17:41.4235816Z c0f36b953c44: Pull complete
2022-12-01T10:17:43.5406298Z 7456bfaa70b2: Pull complete
2022-12-01T10:17:45.7948565Z 68e8ef95b145: Pull complete
2022-12-01T10:17:47.8466375Z e497d45bb922: Pull complete
2022-12-01T10:17:51.6162660Z bf4a66dadd84: Pull complete
2022-12-01T10:17:53.9100092Z fd0d4928961f: Pull complete
2022-12-01T10:17:55.9538973Z 6df8346812f8: Pull complete
2022-12-01T10:18:02.8370818Z 8b46f8aa8681: Pull complete
2022-12-01T10:18:03.6837552Z 6db5688b5394: Pull complete
2022-12-01T10:18:04.1192629Z daa053ac35f3: Pull complete
2022-12-01T10:18:04.2354858Z c9a8257f8dc0: Pull complete
2022-12-01T10:18:04.3626661Z 30f21817172d: Pull complete
2022-12-01T10:18:05.2101250Z bb9282afce05: Pull complete
2022-12-01T10:18:05.3344449Z 909c475d21ec: Pull complete
2022-12-01T10:18:07.2835387Z 192b8944f15a: Pull complete
2022-12-01T10:18:07.3881832Z 3dd92c832839: Pull complete
2022-12-01T10:18:50.0532538Z 5e1b0146a21d: Pull complete
2022-12-01T10:18:51.9644031Z c76cf0339419: Pull complete
2022-12-01T10:18:55.0623547Z 2dc4928eecf0: Pull complete
2022-12-01T10:18:56.4093510Z Digest: sha256:217fd7de680e1dd5bca4b2b4054bd05a8d454df5e210ffbf1e5955e01cf1f340
2022-12-01T10:18:56.9126170Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:18:57.1941743Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:18:57.2042344Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-12-01T10:18:57.2042742Z with:
2022-12-01T10:18:57.2042965Z   timeout_minutes: 10
2022-12-01T10:18:57.2043444Z   max_attempts: 3
2022-12-01T10:18:57.2043846Z   command: set -ex
bash .github/scripts/install_nvidia_utils_linux.sh
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2022-12-01T10:18:57.2044248Z   retry_wait_seconds: 10
2022-12-01T10:18:57.2044510Z   polling_interval_seconds: 1
2022-12-01T10:18:57.2044798Z   warning_on_retry: true
2022-12-01T10:18:57.2045073Z   continue_on_error: false
2022-12-01T10:18:57.2045303Z env:
2022-12-01T10:18:57.2045552Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:18:57.2045821Z ##[endgroup]
2022-12-01T10:18:57.2644588Z 
2022-12-01T10:18:57.2667199Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:18:57.2711404Z + bash .github/scripts/install_nvidia_utils_linux.sh
2022-12-01T10:18:57.2713778Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.57.run ==
2022-12-01T10:18:57.2714808Z + HAS_NVIDIA_DRIVER=0
2022-12-01T10:18:57.2717571Z ++ command -v nvidia-smi
2022-12-01T10:18:57.2720205Z + '[' -x /usr/bin/nvidia-smi ']'
2022-12-01T10:18:57.2723790Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader
2022-12-01T10:19:01.5806439Z + INSTALLED_DRIVER_VERSION='515.76
2022-12-01T10:19:01.5806754Z 515.76'
2022-12-01T10:19:01.5807028Z + '[' '515.76
2022-12-01T10:19:01.5807291Z 515.76' '!=' 515.57 ']'
2022-12-01T10:19:01.5807543Z + HAS_NVIDIA_DRIVER=1
2022-12-01T10:19:01.5807854Z + echo 'NVIDIA driver (515.76
2022-12-01T10:19:01.5808389Z 515.76) has been installed, but we expect to have 515.57 instead. Skipping NVIDIA driver installation for now until torchrec and FBGEMM are updated to use PyTorch NVIDIA installation script instead of RHEL repo'
2022-12-01T10:19:01.5809752Z + '[' 1 -eq 0 ']'
2022-12-01T10:19:01.5810028Z + nvidia-smi
2022-12-01T10:19:01.5810265Z NVIDIA driver (515.76
2022-12-01T10:19:01.5810784Z 515.76) has been installed, but we expect to have 515.57 instead. Skipping NVIDIA driver installation for now until torchrec and FBGEMM are updated to use PyTorch NVIDIA installation script instead of RHEL repo
2022-12-01T10:19:05.4045405Z Thu Dec  1 10:19:05 2022       
2022-12-01T10:19:05.4046425Z +-----------------------------------------------------------------------------+
2022-12-01T10:19:05.4047419Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2022-12-01T10:19:05.4048325Z |-------------------------------+----------------------+----------------------+
2022-12-01T10:19:05.4049265Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2022-12-01T10:19:05.4050251Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2022-12-01T10:19:05.4050991Z |                               |                      |               MIG M. |
2022-12-01T10:19:05.4051524Z |===============================+======================+======================|
2022-12-01T10:19:05.4106627Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |                    0 |
2022-12-01T10:19:05.4107344Z | N/A   26C    P0    38W / 150W |      0MiB /  7680MiB |      0%      Default |
2022-12-01T10:19:05.4108014Z |                               |                      |                  N/A |
2022-12-01T10:19:05.4108826Z +-------------------------------+----------------------+----------------------+
2022-12-01T10:19:05.4212892Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |          12325814270 |
2022-12-01T10:19:05.4213272Z | N/A   34C    P0    38W / 150W |      0MiB /  7680MiB |     66%      Default |
2022-12-01T10:19:05.4213576Z |                               |                      |                  N/A |
2022-12-01T10:19:05.4214322Z +-------------------------------+----------------------+----------------------+
2022-12-01T10:19:05.4214727Z                                                                                
2022-12-01T10:19:05.4215181Z +-----------------------------------------------------------------------------+
2022-12-01T10:19:05.4215674Z | Processes:                                                                  |
2022-12-01T10:19:05.4216030Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2022-12-01T10:19:05.4216373Z |        ID   ID                                                   Usage      |
2022-12-01T10:19:05.4216685Z |=============================================================================|
2022-12-01T10:19:05.4218458Z |  No running processes found                                                 |
2022-12-01T10:19:05.4218950Z +-----------------------------------------------------------------------------+
2022-12-01T10:19:05.4763941Z == Installing nvidia container toolkit for amzn2 ==
2022-12-01T10:19:05.4768168Z + sudo yum install -y yum-utils
2022-12-01T10:19:06.0251694Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:19:07.7835927Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2022-12-01T10:19:07.7836635Z Nothing to do
2022-12-01T10:19:07.8679526Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-12-01T10:19:08.4327618Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:19:08.4657261Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2022-12-01T10:19:08.4657938Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2022-12-01T10:19:08.4658480Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2022-12-01T10:19:08.4805440Z + sudo yum install -y nvidia-docker2
2022-12-01T10:19:09.0329586Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2022-12-01T10:19:10.3696844Z Package nvidia-docker2-2.11.0-1.noarch already installed and latest version
2022-12-01T10:19:10.3697254Z Nothing to do
2022-12-01T10:19:10.4492222Z + sudo systemctl restart docker
2022-12-01T10:19:23.7999002Z + echo 'GPU_FLAG=--gpus all'
2022-12-01T10:19:24.2992792Z Command completed after 1 attempt(s).
2022-12-01T10:19:24.2993286Z 
2022-12-01T10:19:24.2996081Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:19:24.3045488Z ##[group]Run python3 -m pip install psutil==5.9.1
2022-12-01T10:19:24.3045912Z python3 -m pip install psutil==5.9.1
2022-12-01T10:19:24.3046329Z python3 -m pip install pynvml==11.4.1
2022-12-01T10:19:24.3046821Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2022-12-01T10:19:24.3047275Z echo "::set-output name=monitor-script-pid::${!}"
2022-12-01T10:19:24.3088354Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:19:24.3088718Z env:
2022-12-01T10:19:24.3088947Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:24.3089221Z   GPU_FLAG: --gpus all
2022-12-01T10:19:24.3089483Z ##[endgroup]
2022-12-01T10:19:24.6094219Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:19:24.6326091Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2022-12-01T10:19:25.2250189Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T10:19:25.2482251Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2022-12-01T10:19:25.5431899Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:19:25.5479985Z Prepare all required actions
2022-12-01T10:19:25.5480365Z Getting action download info
2022-12-01T10:19:25.7325237Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2022-12-01T10:19:25.9835941Z Download action repository 'actions/download-artifact@v2' (SHA:f023be2c48cc18debc3bacd34cb396e0295e2869)
2022-12-01T10:19:26.1133509Z ##[group]Run ./.github/actions/download-build-artifacts
2022-12-01T10:19:26.1133818Z with:
2022-12-01T10:19:26.1134106Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:19:26.1134369Z env:
2022-12-01T10:19:26.1134610Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:26.1134880Z   GPU_FLAG: --gpus all
2022-12-01T10:19:26.1135113Z ##[endgroup]
2022-12-01T10:19:26.1169183Z ##[group]Run seemethere/download-artifact-s3@v4
2022-12-01T10:19:26.1169482Z with:
2022-12-01T10:19:26.1169744Z   name: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:19:26.1170058Z   s3-bucket: gha-artifacts
2022-12-01T10:19:26.1170368Z   region: us-east-1
2022-12-01T10:19:26.1170611Z env:
2022-12-01T10:19:26.1170849Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:26.1171097Z   GPU_FLAG: --gpus all
2022-12-01T10:19:26.1171361Z ##[endgroup]
2022-12-01T10:19:26.6388860Z Found 1 objects with prefix pytorch/pytorch/3591403534/linux-bionic-cuda11.6-py3.10-gcc7/
2022-12-01T10:19:26.6389786Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-12-01T10:19:35.9084797Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2022-12-01T10:19:35.9085561Z 
2022-12-01T10:19:35.9090874Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:19:35.9092140Z Artifact download has finished successfully
2022-12-01T10:19:35.9407761Z ##[group]Run unzip -o artifacts.zip
2022-12-01T10:19:35.9408091Z unzip -o artifacts.zip
2022-12-01T10:19:35.9422102Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:19:35.9422402Z env:
2022-12-01T10:19:35.9422653Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:35.9422905Z   GPU_FLAG: --gpus all
2022-12-01T10:19:35.9423154Z ##[endgroup]
2022-12-01T10:19:35.9466590Z Archive:  artifacts.zip
2022-12-01T10:19:35.9468668Z    creating: dist/
2022-12-01T10:19:38.0078370Z   inflating: dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl  
2022-12-01T10:19:38.0078855Z    creating: build/custom_test_artifacts/
2022-12-01T10:19:38.0079286Z    creating: build/custom_test_artifacts/custom-op-build/
2022-12-01T10:19:38.0079763Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2022-12-01T10:19:38.0086371Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:19:38.0086925Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2022-12-01T10:19:38.0087504Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:19:38.0088057Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:19:38.0088686Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:19:38.0090948Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:19:38.0092076Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:19:38.0092654Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:19:38.0093216Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:19:38.0096139Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:19:38.0097408Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:19:38.0098832Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:19:38.0099808Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:19:38.0101302Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:19:38.0102468Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:19:38.0103060Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-12-01T10:19:38.0103618Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-12-01T10:19:38.0158349Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-12-01T10:19:38.0159091Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-12-01T10:19:38.0159817Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-12-01T10:19:38.0160553Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-12-01T10:19:38.0161283Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-12-01T10:19:38.0161989Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-12-01T10:19:38.0162687Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-12-01T10:19:38.0163765Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-12-01T10:19:38.0164836Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-12-01T10:19:38.0206902Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-12-01T10:19:38.0248466Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-12-01T10:19:38.0249532Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-12-01T10:19:38.0250289Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-12-01T10:19:38.0251027Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-12-01T10:19:38.0251954Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-12-01T10:19:38.0252986Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-12-01T10:19:38.0253998Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-12-01T10:19:38.0256061Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-12-01T10:19:38.0328984Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-12-01T10:19:38.0402082Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-12-01T10:19:38.0402735Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-12-01T10:19:38.0403302Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2022-12-01T10:19:38.0404184Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2022-12-01T10:19:38.0404802Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:19:38.0405483Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2022-12-01T10:19:38.0406067Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2022-12-01T10:19:38.0406693Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2022-12-01T10:19:38.0407297Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2022-12-01T10:19:38.0407878Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2022-12-01T10:19:38.0408459Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2022-12-01T10:19:38.0409529Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2022-12-01T10:19:38.0410134Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2022-12-01T10:19:38.0410952Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2022-12-01T10:19:38.0411597Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2022-12-01T10:19:38.0432840Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2022-12-01T10:19:38.0547147Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2022-12-01T10:19:38.0547715Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2022-12-01T10:19:38.0548324Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2022-12-01T10:19:38.0548972Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2022-12-01T10:19:38.0549874Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2022-12-01T10:19:38.0550481Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2022-12-01T10:19:38.0551101Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2022-12-01T10:19:38.0552086Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2022-12-01T10:19:38.0553000Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2022-12-01T10:19:38.0553803Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2022-12-01T10:19:38.0554432Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2022-12-01T10:19:38.0575366Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2022-12-01T10:19:38.0657101Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2022-12-01T10:19:38.0657757Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:19:38.0658361Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:19:38.0658935Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2022-12-01T10:19:38.0659696Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2022-12-01T10:19:38.0660953Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:19:38.0661638Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2022-12-01T10:19:38.0664740Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2022-12-01T10:19:38.0665377Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2022-12-01T10:19:38.0666025Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2022-12-01T10:19:38.0759069Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2022-12-01T10:19:38.0821013Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2022-12-01T10:19:38.0821479Z    creating: build/custom_test_artifacts/jit-hook-build/
2022-12-01T10:19:38.0821937Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2022-12-01T10:19:38.0828704Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:19:38.0829247Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2022-12-01T10:19:38.0830041Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:19:38.0830613Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:19:38.0831164Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:19:38.0833319Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:19:38.0834463Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:19:38.0835022Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:19:38.0835586Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:19:38.0838208Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:19:38.0839263Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:19:38.0841425Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:19:38.0842045Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:19:38.0843256Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:19:38.0844201Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:19:38.0844824Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-12-01T10:19:38.0845400Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-12-01T10:19:38.0899906Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-12-01T10:19:38.0900639Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-12-01T10:19:38.0901356Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-12-01T10:19:38.0902101Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-12-01T10:19:38.0902834Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-12-01T10:19:38.0903533Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-12-01T10:19:38.0904211Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-12-01T10:19:38.0904904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-12-01T10:19:38.0905966Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-12-01T10:19:38.0947984Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-12-01T10:19:38.0989134Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-12-01T10:19:38.0990511Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-12-01T10:19:38.0991419Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-12-01T10:19:38.0992070Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-12-01T10:19:38.0992862Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-12-01T10:19:38.0993752Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-12-01T10:19:38.0994733Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-12-01T10:19:38.0996904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-12-01T10:19:38.1069607Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-12-01T10:19:38.1142703Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-12-01T10:19:38.1143341Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-12-01T10:19:38.1143881Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2022-12-01T10:19:38.1144689Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2022-12-01T10:19:38.1145257Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:19:38.1145803Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2022-12-01T10:19:38.1146382Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2022-12-01T10:19:38.1147175Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2022-12-01T10:19:38.1147806Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2022-12-01T10:19:38.1148394Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2022-12-01T10:19:38.1149144Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2022-12-01T10:19:38.1150582Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2022-12-01T10:19:38.1151245Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2022-12-01T10:19:38.1151983Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2022-12-01T10:19:38.1153315Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2022-12-01T10:19:38.1174357Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2022-12-01T10:19:38.1238029Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2022-12-01T10:19:38.1238651Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:19:38.1239260Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:19:38.1239824Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2022-12-01T10:19:38.1240721Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2022-12-01T10:19:38.1241925Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:19:38.1242539Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2022-12-01T10:19:38.1245441Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2022-12-01T10:19:38.1245990Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2022-12-01T10:19:38.1246957Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2022-12-01T10:19:38.1296343Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2022-12-01T10:19:38.1296829Z    creating: build/custom_test_artifacts/custom-backend-build/
2022-12-01T10:19:38.1297320Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2022-12-01T10:19:38.1304025Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2022-12-01T10:19:38.1304609Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2022-12-01T10:19:38.1305184Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2022-12-01T10:19:38.1305780Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2022-12-01T10:19:38.1306345Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2022-12-01T10:19:38.1308660Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2022-12-01T10:19:38.1310167Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2022-12-01T10:19:38.1310764Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2022-12-01T10:19:38.1311340Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2022-12-01T10:19:38.1313674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2022-12-01T10:19:38.1314829Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2022-12-01T10:19:38.1316773Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2022-12-01T10:19:38.1317411Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2022-12-01T10:19:38.1318892Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2022-12-01T10:19:38.1320138Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2022-12-01T10:19:38.1320753Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2022-12-01T10:19:38.1321350Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2022-12-01T10:19:38.1375667Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2022-12-01T10:19:38.1376415Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2022-12-01T10:19:38.1377165Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2022-12-01T10:19:38.1377927Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2022-12-01T10:19:38.1378661Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2022-12-01T10:19:38.1379385Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2022-12-01T10:19:38.1380216Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2022-12-01T10:19:38.1380970Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2022-12-01T10:19:38.1381846Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2022-12-01T10:19:38.1424107Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2022-12-01T10:19:38.1465389Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2022-12-01T10:19:38.1466423Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2022-12-01T10:19:38.1467157Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2022-12-01T10:19:38.1467814Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2022-12-01T10:19:38.1468729Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2022-12-01T10:19:38.1469967Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2022-12-01T10:19:38.1471011Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2022-12-01T10:19:38.1473052Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2022-12-01T10:19:38.1545735Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2022-12-01T10:19:38.1618998Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2022-12-01T10:19:38.1619669Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2022-12-01T10:19:38.1620253Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2022-12-01T10:19:38.1620968Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2022-12-01T10:19:38.1621555Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2022-12-01T10:19:38.1622130Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2022-12-01T10:19:38.1622902Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2022-12-01T10:19:38.1623563Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2022-12-01T10:19:38.1624188Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2022-12-01T10:19:38.1625270Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2022-12-01T10:19:38.1625948Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2022-12-01T10:19:38.1626864Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2022-12-01T10:19:38.1627521Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2022-12-01T10:19:38.1628163Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2022-12-01T10:19:38.1628790Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2022-12-01T10:19:38.1633928Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2022-12-01T10:19:38.1781206Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2022-12-01T10:19:38.1781879Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2022-12-01T10:19:38.1782634Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2022-12-01T10:19:38.1783303Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2022-12-01T10:19:38.1783957Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2022-12-01T10:19:38.1784595Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2022-12-01T10:19:38.1785242Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2022-12-01T10:19:38.1785877Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2022-12-01T10:19:38.1786536Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2022-12-01T10:19:38.1787188Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2022-12-01T10:19:38.1787826Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2022-12-01T10:19:38.1808676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2022-12-01T10:19:38.1866567Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2022-12-01T10:19:38.1867252Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2022-12-01T10:19:38.1867898Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2022-12-01T10:19:38.1868483Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2022-12-01T10:19:38.1869245Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2022-12-01T10:19:38.1870726Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2022-12-01T10:19:38.1871290Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2022-12-01T10:19:38.1874268Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2022-12-01T10:19:38.1875022Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2022-12-01T10:19:38.1875853Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2022-12-01T10:19:38.1995008Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2022-12-01T10:19:38.2040616Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2022-12-01T10:19:38.2040990Z    creating: build/lib/
2022-12-01T10:19:38.2041633Z   inflating: build/lib/libclog.a     
2022-12-01T10:19:38.2107761Z   inflating: build/lib/libgtest.a    
2022-12-01T10:19:38.2118288Z   inflating: build/lib/libpthreadpool.a  
2022-12-01T10:19:38.2224187Z   inflating: build/lib/libprotobuf-lite.a  
2022-12-01T10:19:38.2315994Z   inflating: build/lib/libbenchmark.a  
2022-12-01T10:19:38.2325085Z   inflating: build/lib/libittnotify.a  
2022-12-01T10:19:38.2401979Z   inflating: build/lib/libasmjit.a   
2022-12-01T10:19:38.2935794Z   inflating: build/lib/libprotobuf.a  
2022-12-01T10:19:38.2967889Z   inflating: build/lib/libtensorpipe_uv.a  
2022-12-01T10:19:38.3100476Z   inflating: build/lib/libgloo.a     
2022-12-01T10:19:38.3120300Z   inflating: build/lib/libfmt.a      
2022-12-01T10:19:38.3122149Z   inflating: build/lib/libcaffe2_nvrtc.so  
2022-12-01T10:19:38.3122679Z   inflating: build/lib/libfoxi_loader.a  
2022-12-01T10:19:38.3199767Z   inflating: build/lib/libc10.so     
2022-12-01T10:19:38.3200997Z   inflating: build/lib/libtorch_global_deps.so  
2022-12-01T10:19:38.3211043Z   inflating: build/lib/libcpuinfo.a  
2022-12-01T10:19:38.3780718Z   inflating: build/lib/libprotoc.a   
2022-12-01T10:19:38.3789623Z   inflating: build/lib/libcpuinfo_internals.a  
2022-12-01T10:19:38.3805846Z   inflating: build/lib/libqnnpack.a  
2022-12-01T10:19:38.3829690Z   inflating: build/lib/libpytorch_qnnpack.a  
2022-12-01T10:19:38.3832485Z   inflating: build/lib/libnnpack_reference_layers.a  
2022-12-01T10:19:38.3854519Z   inflating: build/lib/libnnpack.a   
2022-12-01T10:19:38.3873650Z   inflating: build/lib/libgmock.a    
2022-12-01T10:19:38.3874234Z   inflating: build/lib/libgtest_main.a  
2022-12-01T10:19:38.3875283Z   inflating: build/lib/libbenchmark_main.a  
2022-12-01T10:19:39.1990227Z   inflating: build/lib/libdnnl.a     
2022-12-01T10:19:39.2131630Z   inflating: build/lib/libXNNPACK.a  
2022-12-01T10:19:39.2787060Z   inflating: build/lib/libtensorpipe.a  
2022-12-01T10:19:39.2829062Z   inflating: build/lib/libc10_cuda.so  
2022-12-01T10:19:39.4369552Z   inflating: build/lib/libfbgemm.a   
2022-12-01T10:19:39.4370292Z   inflating: build/lib/libgmock_main.a  
2022-12-01T10:19:39.5499795Z   inflating: build/lib/libdnnl_graph.a  
2022-12-01T10:19:39.5921126Z   inflating: build/lib/libkineto.a   
2022-12-01T10:19:39.6210868Z   inflating: build/lib/libtensorpipe_cuda.a  
2022-12-01T10:19:39.6256211Z   inflating: build/lib/libcaffe2_protos.a  
2022-12-01T10:19:39.6304156Z   inflating: build/lib/libonnx_proto.a  
2022-12-01T10:19:39.6982404Z   inflating: build/lib/libonnx.a     
2022-12-01T10:19:39.7411195Z   inflating: build/lib/libgloo_cuda.a  
2022-12-01T10:19:42.0600021Z   inflating: build/lib/libtorch_cpu.so  
2022-12-01T10:19:42.4028057Z   inflating: build/lib/libtorch_cuda_cpp.so  
2022-12-01T10:19:44.1684488Z   inflating: build/lib/libtorch_cuda_cu.so  
2022-12-01T10:19:44.1685237Z   inflating: build/lib/libtorch_cuda.so  
2022-12-01T10:19:44.1686981Z   inflating: build/lib/libtorch.so   
2022-12-01T10:19:45.1567480Z   inflating: build/lib/libtorch_cuda_linalg.so  
2022-12-01T10:19:45.1570057Z   inflating: build/lib/libc10d_cuda_test.so  
2022-12-01T10:19:45.1595022Z   inflating: build/lib/libjitbackend_test.so  
2022-12-01T10:19:45.1655158Z   inflating: build/lib/libtorchbind_test.so  
2022-12-01T10:19:45.1686052Z   inflating: build/lib/libbackend_with_compiler.so  
2022-12-01T10:19:45.1690768Z   inflating: build/lib/libshm.so     
2022-12-01T10:19:45.3469841Z   inflating: build/lib/libtorch_python.so  
2022-12-01T10:19:45.3509353Z   inflating: build/lib/libnnapi_backend.so  
2022-12-01T10:19:45.3509663Z    creating: build/bin/
2022-12-01T10:19:45.3562299Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2022-12-01T10:19:45.3617417Z   inflating: build/bin/c10_DeviceGuard_test  
2022-12-01T10:19:45.3671174Z   inflating: build/bin/c10_Device_test  
2022-12-01T10:19:45.3734162Z   inflating: build/bin/c10_DispatchKeySet_test  
2022-12-01T10:19:45.3785310Z   inflating: build/bin/c10_StreamGuard_test  
2022-12-01T10:19:45.3839007Z   inflating: build/bin/c10_SymInt_test  
2022-12-01T10:19:45.3898327Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2022-12-01T10:19:45.3958027Z   inflating: build/bin/c10_InlineStreamGuard_test  
2022-12-01T10:19:45.4018873Z   inflating: build/bin/c10_SizesAndStrides_test  
2022-12-01T10:19:45.4070001Z   inflating: build/bin/c10_Array_test  
2022-12-01T10:19:45.4126777Z   inflating: build/bin/c10_Bitset_test  
2022-12-01T10:19:45.4178145Z   inflating: build/bin/c10_ConstexprCrc_test  
2022-12-01T10:19:45.4233106Z   inflating: build/bin/c10_C++17_test  
2022-12-01T10:19:45.4285430Z   inflating: build/bin/c10_DeadlockDetection_test  
2022-12-01T10:19:45.4338274Z   inflating: build/bin/c10_Half_test  
2022-12-01T10:19:45.4398924Z   inflating: build/bin/c10_LeftRight_test  
2022-12-01T10:19:45.4465916Z   inflating: build/bin/c10_Metaprogramming_test  
2022-12-01T10:19:45.4621811Z   inflating: build/bin/c10_SmallVectorTest  
2022-12-01T10:19:45.4675720Z   inflating: build/bin/c10_Synchronized_test  
2022-12-01T10:19:45.4736780Z   inflating: build/bin/c10_ThreadLocal_test  
2022-12-01T10:19:45.4792843Z   inflating: build/bin/c10_TypeIndex_test  
2022-12-01T10:19:45.4844689Z   inflating: build/bin/c10_TypeTraits_test  
2022-12-01T10:19:45.4899585Z   inflating: build/bin/c10_accumulate_test  
2022-12-01T10:19:45.4953613Z   inflating: build/bin/c10_TypeList_test  
2022-12-01T10:19:45.5013701Z   inflating: build/bin/c10_bfloat16_test  
2022-12-01T10:19:45.5071382Z   inflating: build/bin/c10_complex_math_test  
2022-12-01T10:19:45.5130671Z   inflating: build/bin/c10_complex_test  
2022-12-01T10:19:45.5249160Z   inflating: build/bin/c10_either_test  
2022-12-01T10:19:45.5304682Z   inflating: build/bin/c10_exception_test  
2022-12-01T10:19:45.5358208Z   inflating: build/bin/c10_flags_test  
2022-12-01T10:19:45.5412068Z   inflating: build/bin/c10_irange_test  
2022-12-01T10:19:45.5594940Z   inflating: build/bin/c10_intrusive_ptr_test  
2022-12-01T10:19:45.5656493Z   inflating: build/bin/c10_logging_test  
2022-12-01T10:19:45.5736389Z   inflating: build/bin/c10_optional_test  
2022-12-01T10:19:45.5802792Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2022-12-01T10:19:45.5861009Z   inflating: build/bin/c10_registry_test  
2022-12-01T10:19:45.5924684Z   inflating: build/bin/c10_string_view_test  
2022-12-01T10:19:45.5979458Z   inflating: build/bin/c10_tempfile_test  
2022-12-01T10:19:45.6039994Z   inflating: build/bin/c10_typeid_test  
2022-12-01T10:19:45.6100145Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2022-12-01T10:19:45.6621768Z   inflating: build/bin/protoc-3.13.0.0  
2022-12-01T10:19:45.7144059Z   inflating: build/bin/protoc        
2022-12-01T10:19:45.7195808Z   inflating: build/bin/c10_cuda_CUDATest  
2022-12-01T10:19:45.7513526Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2022-12-01T10:19:45.7867739Z   inflating: build/bin/vec_test_all_types_AVX2  
2022-12-01T10:19:45.7925530Z   inflating: build/bin/HashStoreTest  
2022-12-01T10:19:45.7990028Z   inflating: build/bin/TCPStoreTest  
2022-12-01T10:19:45.8047576Z   inflating: build/bin/FileStoreTest  
2022-12-01T10:19:45.8063346Z   inflating: build/bin/ProcessGroupMPITest  
2022-12-01T10:19:45.8066578Z   inflating: build/bin/example_allreduce  
2022-12-01T10:19:45.8122679Z   inflating: build/bin/Dimname_test  
2022-12-01T10:19:45.8201029Z   inflating: build/bin/Dict_test     
2022-12-01T10:19:45.8269105Z   inflating: build/bin/MaybeOwned_test  
2022-12-01T10:19:45.8332481Z   inflating: build/bin/apply_utils_test  
2022-12-01T10:19:45.8393627Z   inflating: build/bin/NamedTensor_test  
2022-12-01T10:19:45.8456761Z   inflating: build/bin/atest         
2022-12-01T10:19:45.8514755Z   inflating: build/bin/broadcast_test  
2022-12-01T10:19:45.8580310Z   inflating: build/bin/basic         
2022-12-01T10:19:45.8642976Z   inflating: build/bin/cpu_generator_test  
2022-12-01T10:19:45.8698792Z   inflating: build/bin/cpu_profiling_allocator_test  
2022-12-01T10:19:45.8752137Z   inflating: build/bin/dispatch_key_set_test  
2022-12-01T10:19:45.8847451Z   inflating: build/bin/cpu_rng_test  
2022-12-01T10:19:45.8900381Z   inflating: build/bin/dlconvertor_test  
2022-12-01T10:19:45.8962788Z   inflating: build/bin/extension_backend_test  
2022-12-01T10:19:45.9022450Z   inflating: build/bin/half_test     
2022-12-01T10:19:45.9075046Z   inflating: build/bin/lazy_tensor_test  
2022-12-01T10:19:45.9176645Z   inflating: build/bin/ivalue_test   
2022-12-01T10:19:45.9233976Z   inflating: build/bin/math_kernel_test  
2022-12-01T10:19:45.9291001Z   inflating: build/bin/memory_format_test  
2022-12-01T10:19:45.9347211Z   inflating: build/bin/memory_overlapping_test  
2022-12-01T10:19:45.9401823Z   inflating: build/bin/operator_name_test  
2022-12-01T10:19:45.9461818Z   inflating: build/bin/native_test   
2022-12-01T10:19:45.9517542Z   inflating: build/bin/mobile_memory_cleanup  
2022-12-01T10:19:45.9570986Z   inflating: build/bin/operators_test  
2022-12-01T10:19:45.9627022Z   inflating: build/bin/packedtensoraccessor_test  
2022-12-01T10:19:45.9697312Z   inflating: build/bin/pow_test      
2022-12-01T10:19:45.9758694Z   inflating: build/bin/quantized_test  
2022-12-01T10:19:45.9812813Z   inflating: build/bin/reportMemoryUsage_test  
2022-12-01T10:19:45.9865439Z   inflating: build/bin/reduce_ops_test  
2022-12-01T10:19:45.9925896Z   inflating: build/bin/scalar_tensor_test  
2022-12-01T10:19:45.9988585Z   inflating: build/bin/scalar_test   
2022-12-01T10:19:46.0044339Z   inflating: build/bin/stride_properties_test  
2022-12-01T10:19:46.0046890Z   inflating: build/bin/thread_init_test  
2022-12-01T10:19:46.0131329Z   inflating: build/bin/tensor_iterator_test  
2022-12-01T10:19:46.0190800Z   inflating: build/bin/type_ptr_test  
2022-12-01T10:19:46.0250618Z   inflating: build/bin/test_parallel  
2022-12-01T10:19:46.0315312Z   inflating: build/bin/type_test     
2022-12-01T10:19:46.0368128Z   inflating: build/bin/variant_test  
2022-12-01T10:19:46.0424722Z   inflating: build/bin/undefined_tensor_test  
2022-12-01T10:19:46.0499062Z   inflating: build/bin/vmap_test     
2022-12-01T10:19:46.0500442Z   inflating: build/bin/verify_api_visibility  
2022-12-01T10:19:46.0555202Z   inflating: build/bin/weakref_test  
2022-12-01T10:19:46.0609403Z   inflating: build/bin/wrapdim_test  
2022-12-01T10:19:46.0673555Z   inflating: build/bin/IListRef_test  
2022-12-01T10:19:46.0725605Z   inflating: build/bin/xla_tensor_test  
2022-12-01T10:19:46.0843876Z   inflating: build/bin/List_test     
2022-12-01T10:19:46.0913804Z   inflating: build/bin/KernelFunction_test  
2022-12-01T10:19:46.1045032Z   inflating: build/bin/kernel_function_legacy_test  
2022-12-01T10:19:46.1150004Z   inflating: build/bin/kernel_function_test  
2022-12-01T10:19:46.1289084Z   inflating: build/bin/kernel_lambda_legacy_test  
2022-12-01T10:19:46.1401125Z   inflating: build/bin/kernel_lambda_test  
2022-12-01T10:19:46.1465230Z   inflating: build/bin/kernel_stackbased_test  
2022-12-01T10:19:46.1568697Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2022-12-01T10:19:46.1622425Z   inflating: build/bin/CppSignature_test  
2022-12-01T10:19:46.1674004Z   inflating: build/bin/op_allowlist_test  
2022-12-01T10:19:46.1730229Z   inflating: build/bin/inline_container_test  
2022-12-01T10:19:46.2043864Z   inflating: build/bin/op_registration_test  
2022-12-01T10:19:46.2104116Z   inflating: build/bin/backend_fallback_test  
2022-12-01T10:19:46.2159901Z   inflating: build/bin/cuda_apply_test  
2022-12-01T10:19:46.2217074Z   inflating: build/bin/cuda_caching_host_allocator_test  
2022-12-01T10:19:46.2281881Z   inflating: build/bin/cuda_atomic_ops_test  
2022-12-01T10:19:46.2354856Z   inflating: build/bin/cuda_complex_math_test  
2022-12-01T10:19:46.2406988Z   inflating: build/bin/cuda_device_test  
2022-12-01T10:19:46.2469582Z   inflating: build/bin/cuda_complex_test  
2022-12-01T10:19:46.2533507Z   inflating: build/bin/cuda_cub_test  
2022-12-01T10:19:46.2586673Z   inflating: build/bin/cuda_dlconvertor_test  
2022-12-01T10:19:46.2640951Z   inflating: build/bin/cuda_integer_divider_test  
2022-12-01T10:19:46.2713059Z   inflating: build/bin/cuda_distributions_test  
2022-12-01T10:19:46.2775548Z   inflating: build/bin/cuda_generator_test  
2022-12-01T10:19:46.2828320Z   inflating: build/bin/cuda_half_test  
2022-12-01T10:19:46.2880360Z   inflating: build/bin/cuda_optional_test  
2022-12-01T10:19:46.2936374Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2022-12-01T10:19:46.3001402Z   inflating: build/bin/cuda_stream_test  
2022-12-01T10:19:46.3056057Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2022-12-01T10:19:46.3107750Z   inflating: build/bin/cuda_cudnn_test  
2022-12-01T10:19:46.3163774Z   inflating: build/bin/cuda_vectorized_test  
2022-12-01T10:19:46.3181365Z   inflating: build/bin/tutorial_tensorexpr  
2022-12-01T10:19:46.3250963Z   inflating: build/bin/ProcessGroupGlooTest  
2022-12-01T10:19:46.3313480Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2022-12-01T10:19:46.3379205Z   inflating: build/bin/ProcessGroupNCCLTest  
2022-12-01T10:19:46.3441783Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2022-12-01T10:19:46.3498228Z   inflating: build/bin/ProcessGroupUCCTest  
2022-12-01T10:19:46.3556248Z   inflating: build/bin/test_dist_autograd  
2022-12-01T10:19:46.3559029Z   inflating: build/bin/parallel_benchmark  
2022-12-01T10:19:46.3634004Z   inflating: build/bin/test_cpp_rpc  
2022-12-01T10:19:46.3707786Z   inflating: build/bin/test_mobile_nnc  
2022-12-01T10:19:46.4628122Z   inflating: build/bin/test_tensorexpr  
2022-12-01T10:19:46.4639871Z   inflating: build/bin/aot_model_compiler_test  
2022-12-01T10:19:46.5023827Z   inflating: build/bin/test_lazy     
2022-12-01T10:19:46.5029417Z   inflating: build/bin/torch_shm_manager  
2022-12-01T10:19:46.6350265Z   inflating: build/bin/test_api      
2022-12-01T10:19:46.7519311Z   inflating: build/bin/test_jit      
2022-12-01T10:19:46.7521497Z   inflating: .pytorch-test-times.json  
2022-12-01T10:19:46.7553725Z ##[group]Run df -H
2022-12-01T10:19:46.7553984Z df -H
2022-12-01T10:19:46.7567656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T10:19:46.7567962Z env:
2022-12-01T10:19:46.7568203Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:46.7568457Z   GPU_FLAG: --gpus all
2022-12-01T10:19:46.7568718Z ##[endgroup]
2022-12-01T10:19:46.7609192Z Filesystem      Size  Used Avail Use% Mounted on
2022-12-01T10:19:46.7609531Z devtmpfs        129G     0  129G   0% /dev
2022-12-01T10:19:46.7609841Z tmpfs           129G  6.3M  129G   1% /dev/shm
2022-12-01T10:19:46.7610105Z tmpfs           129G  553k  129G   1% /run
2022-12-01T10:19:46.7610398Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2022-12-01T10:19:46.7610945Z /dev/xvda1      162G   30G  132G  19% /
2022-12-01T10:19:46.7636709Z ##[group]Run .github/scripts/parse_ref.py
2022-12-01T10:19:46.7637075Z .github/scripts/parse_ref.py
2022-12-01T10:19:46.7649026Z shell: /usr/bin/bash -e {0}
2022-12-01T10:19:46.7649283Z env:
2022-12-01T10:19:46.7649506Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:46.7649781Z   GPU_FLAG: --gpus all
2022-12-01T10:19:46.7650029Z ##[endgroup]
2022-12-01T10:19:46.7892460Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T10:19:46.7953697Z ##[group]Run set -x
2022-12-01T10:19:46.7954082Z set -x
2022-12-01T10:19:46.7954313Z 
2022-12-01T10:19:46.7954565Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2022-12-01T10:19:46.7954913Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2022-12-01T10:19:46.7955262Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2022-12-01T10:19:46.7955567Z   TEST_COMMAND=.jenkins/caffe2/test.sh
2022-12-01T10:19:46.7955838Z else
2022-12-01T10:19:46.7956117Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2022-12-01T10:19:46.7956372Z fi
2022-12-01T10:19:46.7956587Z 
2022-12-01T10:19:46.7956907Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2022-12-01T10:19:46.7957207Z 
2022-12-01T10:19:46.7957502Z # sanitize the input commit message and PR body here:
2022-12-01T10:19:46.7957804Z #
2022-12-01T10:19:46.7958160Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2022-12-01T10:19:46.7958662Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2022-12-01T10:19:46.7959082Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2022-12-01T10:19:46.7959395Z PR_BODY="${PR_BODY//[$'\n\r']}"
2022-12-01T10:19:46.7959633Z 
2022-12-01T10:19:46.7959986Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2022-12-01T10:19:46.7960359Z # wreak havoc internally
2022-12-01T10:19:46.7960663Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2022-12-01T10:19:46.7960993Z export PR_BODY="${PR_BODY//[\'\"]}"
2022-12-01T10:19:46.7961255Z 
2022-12-01T10:19:46.7961562Z # detached container should get cleaned up by teardown_ec2_linux
2022-12-01T10:19:46.7962093Z # TODO: Stop building test binaries as part of the build phase
2022-12-01T10:19:46.7962470Z # Used for GPU_FLAG since that doesn't play nice
2022-12-01T10:19:46.7962797Z # shellcheck disable=SC2086,SC2090
2022-12-01T10:19:46.7963080Z container_name=$(docker run \
2022-12-01T10:19:46.7963355Z   ${GPU_FLAG:-} \
2022-12-01T10:19:46.7963623Z   -e BUILD_ENVIRONMENT \
2022-12-01T10:19:46.7963878Z   -e PR_NUMBER \
2022-12-01T10:19:46.7964141Z   -e GITHUB_ACTIONS \
2022-12-01T10:19:46.7964398Z   -e BASE_SHA \
2022-12-01T10:19:46.7964645Z   -e BRANCH \
2022-12-01T10:19:46.7964869Z   -e SHA1 \
2022-12-01T10:19:46.7965126Z   -e AWS_DEFAULT_REGION \
2022-12-01T10:19:46.7965395Z   -e IN_WHEEL_TEST \
2022-12-01T10:19:46.7965639Z   -e SHARD_NUMBER \
2022-12-01T10:19:46.7965896Z   -e TEST_CONFIG \
2022-12-01T10:19:46.7966164Z   -e NUM_TEST_SHARDS \
2022-12-01T10:19:46.7966405Z   -e PR_BODY \
2022-12-01T10:19:46.7966666Z   -e COMMIT_MESSAGES \
2022-12-01T10:19:46.7966952Z   -e PYTORCH_RETRY_TEST_CASES \
2022-12-01T10:19:46.7967247Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2022-12-01T10:19:46.7967535Z   -e PR_LABELS \
2022-12-01T10:19:46.7967820Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2022-12-01T10:19:46.7968092Z   -e SCCACHE_BUCKET \
2022-12-01T10:19:46.7968370Z   -e SCCACHE_S3_KEY_PREFIX \
2022-12-01T10:19:46.7968639Z   -e XLA_CUDA \
2022-12-01T10:19:46.7968923Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2022-12-01T10:19:46.7969246Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2022-12-01T10:19:46.7969567Z   --ulimit stack=10485760:83886080 \
2022-12-01T10:19:46.7969880Z   --security-opt seccomp=unconfined \
2022-12-01T10:19:46.7970171Z   --cap-add=SYS_PTRACE \
2022-12-01T10:19:46.7970444Z   --ipc=host \
2022-12-01T10:19:46.7970786Z   --shm-size="${SHM_SIZE}" \
2022-12-01T10:19:46.7971042Z   --tty \
2022-12-01T10:19:46.7971283Z   --detach \
2022-12-01T10:19:46.7971549Z   --name="${container_name}" \
2022-12-01T10:19:46.7971806Z   --user jenkins \
2022-12-01T10:19:46.7972127Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2022-12-01T10:19:46.7972472Z   -w /var/lib/jenkins/workspace \
2022-12-01T10:19:46.7972754Z   "${DOCKER_IMAGE}"
2022-12-01T10:19:46.7972977Z )
2022-12-01T10:19:46.7973345Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2022-12-01T10:19:46.7986028Z shell: /usr/bin/bash -e {0}
2022-12-01T10:19:46.7986279Z env:
2022-12-01T10:19:46.7986521Z   GIT_DEFAULT_BRANCH: master
2022-12-01T10:19:46.7986767Z   GPU_FLAG: --gpus all
2022-12-01T10:19:46.7987099Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:19:46.7987429Z   PR_NUMBER: 89997
2022-12-01T10:19:46.7987663Z   BRANCH: pull/89997
2022-12-01T10:19:46.7987956Z   SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:19:46.7988354Z   BASE_SHA: ae2fe4033cf3b17259b17f351020b988fa893f91
2022-12-01T10:19:46.7988647Z   PYTORCH_RETRY_TEST_CASES: 1
2022-12-01T10:19:46.7988934Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-12-01T10:19:46.7989214Z   TEST_CONFIG: distributed
2022-12-01T10:19:46.7989897Z   SHARD_NUMBER: 1
2022-12-01T10:19:46.7990132Z   NUM_TEST_SHARDS: 3
2022-12-01T10:19:46.7990996Z   PR_BODY: Link to landed master PR (if applicable):
https://github.com/pytorch/pytorch/pull/88993

Criteria category:
1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.13
2: Prevents a crash

I'm hoping this is a low-risk change, since it's just changing one method for its safer form.
2022-12-01T10:19:46.7992025Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2022-12-01T10:19:46.7992357Z   SCCACHE_S3_KEY_PREFIX: pull
2022-12-01T10:19:46.7992595Z   SHM_SIZE: 2g
2022-12-01T10:19:46.7993083Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:19:46.7993548Z   XLA_CUDA: 
2022-12-01T10:19:46.7993902Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2022-12-01T10:19:46.7994236Z ##[endgroup]
2022-12-01T10:19:46.8022848Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2022-12-01T10:19:46.8023320Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *onnx* ]]
2022-12-01T10:19:46.8023666Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2022-12-01T10:19:46.8026703Z ++ git cherry -v origin/master
2022-12-01T10:19:47.2578658Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)
2022-12-01T10:19:47.2579310Z - f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)
2022-12-01T10:19:47.2579899Z + 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)
2022-12-01T10:19:47.2580560Z - c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)
2022-12-01T10:19:47.2581190Z - 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)
2022-12-01T10:19:47.2581803Z - 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)
2022-12-01T10:19:47.2582407Z - f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)
2022-12-01T10:19:47.2583002Z - 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)
2022-12-01T10:19:47.2583609Z - 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)
2022-12-01T10:19:47.2584259Z + d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)
2022-12-01T10:19:47.2584842Z + 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)
2022-12-01T10:19:47.2585390Z - 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)
2022-12-01T10:19:47.2585831Z + f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)
2022-12-01T10:19:47.2586459Z - 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)
2022-12-01T10:19:47.2587114Z - 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)
2022-12-01T10:19:47.2587646Z + 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)
2022-12-01T10:19:47.2588337Z - aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)
2022-12-01T10:19:47.2588953Z - de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)
2022-12-01T10:19:47.2589902Z + 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)
2022-12-01T10:19:47.2590367Z + 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)
2022-12-01T10:19:47.2591372Z - f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)
2022-12-01T10:19:47.2592060Z - c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)
2022-12-01T10:19:47.2592566Z + eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)
2022-12-01T10:19:47.2593043Z + d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)
2022-12-01T10:19:47.2593782Z - 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)
2022-12-01T10:19:47.2594385Z - 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)
2022-12-01T10:19:47.2594984Z - 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)
2022-12-01T10:19:47.2595440Z + d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)
2022-12-01T10:19:47.2596067Z - 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)
2022-12-01T10:19:47.2596691Z - ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)
2022-12-01T10:19:47.2597288Z - e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)
2022-12-01T10:19:47.2597809Z + 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)
2022-12-01T10:19:47.2598431Z - 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)
2022-12-01T10:19:47.2599056Z - 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)
2022-12-01T10:19:47.2599625Z - 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)
2022-12-01T10:19:47.2600242Z - 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)
2022-12-01T10:19:47.2600896Z - 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)
2022-12-01T10:19:47.2601471Z - aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)
2022-12-01T10:19:47.2602183Z - 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)
2022-12-01T10:19:47.2602748Z + ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)
2022-12-01T10:19:47.2603396Z - d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)
2022-12-01T10:19:47.2603998Z - 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)
2022-12-01T10:19:47.2604554Z - 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)
2022-12-01T10:19:47.2605086Z - 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)
2022-12-01T10:19:47.2605536Z + 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)
2022-12-01T10:19:47.2606062Z - 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)
2022-12-01T10:19:47.2606514Z + 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)
2022-12-01T10:19:47.2606956Z + d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)
2022-12-01T10:19:47.2621104Z - d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)
2022-12-01T10:19:47.2621724Z - 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)
2022-12-01T10:19:47.2622352Z - f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)
2022-12-01T10:19:47.2623022Z - 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)
2022-12-01T10:19:47.2623528Z + 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)
2022-12-01T10:19:47.2624116Z - fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)
2022-12-01T10:19:47.2624821Z - 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)
2022-12-01T10:19:47.2625340Z - 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)
2022-12-01T10:19:47.2625891Z - 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)
2022-12-01T10:19:47.2626488Z - 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)
2022-12-01T10:19:47.2626909Z + ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)
2022-12-01T10:19:47.2627452Z - c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:19:47.2643302Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:19:47.2651864Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashI'\''m hoping this is a low-risk change, since it'\''s just changing one method for its safer form.'
2022-12-01T10:19:47.2667723Z + export 'COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:19:47.2690447Z + COMMIT_MESSAGES='+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack'
2022-12-01T10:19:47.2698899Z + export 'PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.'
2022-12-01T10:19:47.2700778Z + PR_BODY='Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.'
2022-12-01T10:19:47.2701613Z +++ nproc --ignore=2
2022-12-01T10:19:47.2703617Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME --env-file=/tmp/github_env_3591403534 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T10:20:00.9531665Z + container_name=7ff5451e42d4638cf6b6098bb9ca823aa5cb3bfb8c9e9a1aea66c1e90c6dce30
2022-12-01T10:20:00.9535947Z ++ echo dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl
2022-12-01T10:20:00.9538296Z + docker exec -t 7ff5451e42d4638cf6b6098bb9ca823aa5cb3bfb8c9e9a1aea66c1e90c6dce30 sh -c 'pip install dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2022-12-01T10:20:01.5389710Z Processing ./dist/torch-1.13.0a0+gitc13d400-cp310-cp310-linux_x86_64.whl
2022-12-01T10:20:01.6332532Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+gitc13d400) (4.3.0)
2022-12-01T10:20:01.6346203Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==1.13.0a0+gitc13d400) (3.3.0)
2022-12-01T10:20:01.6427097Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==1.13.0a0+gitc13d400) (1.21.2)
2022-12-01T10:20:02.5785029Z Installing collected packages: torch
2022-12-01T10:20:13.3221212Z Successfully installed torch-1.13.0a0+gitc13d400
2022-12-01T10:20:13.4936368Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2022-12-01T10:20:13.5156391Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2022-12-01T10:20:13.5160040Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T10:20:13.5161026Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2022-12-01T10:20:13.5161698Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2022-12-01T10:20:13.5162059Z + BUILD_DIR=build
2022-12-01T10:20:13.5162496Z + BUILD_RENAMED_DIR=build_renamed
2022-12-01T10:20:13.5162832Z + BUILD_BIN_DIR=build/bin
2022-12-01T10:20:13.5163220Z + export VALGRIND=ON
2022-12-01T10:20:13.5163526Z + VALGRIND=ON
2022-12-01T10:20:13.5163981Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *clang9* ]]
2022-12-01T10:20:13.5164435Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *bazel* ]]
2022-12-01T10:20:13.5164875Z ++ realpath build/custom_test_artifacts
2022-12-01T10:20:13.5170453Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2022-12-01T10:20:13.5173421Z ++ dirname .jenkins/pytorch/test.sh
2022-12-01T10:20:13.5180925Z + source .jenkins/pytorch/common.sh
2022-12-01T10:20:13.5184918Z +++ dirname .jenkins/pytorch/common.sh
2022-12-01T10:20:13.5196741Z ++ source .jenkins/pytorch/common_utils.sh
2022-12-01T10:20:13.5198897Z +++ declare -f -t trap_add
2022-12-01T10:20:13.5203039Z ++ set -ex
2022-12-01T10:20:13.5203755Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-12-01T10:20:13.5204353Z ++ BUILD_TEST_LIBTORCH=0
2022-12-01T10:20:13.5204687Z ++ [[ distributed == *xla* ]]
2022-12-01T10:20:13.5205166Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-12-01T10:20:13.5205694Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *linux-bionic* ]]
2022-12-01T10:20:13.5206053Z ++ which conda
2022-12-01T10:20:13.5214471Z /opt/conda/bin/conda
2022-12-01T10:20:13.5263258Z ++ conda install -q -y cmake
2022-12-01T10:20:17.0884146Z Collecting package metadata (current_repodata.json): ...working... done
2022-12-01T10:20:18.3452607Z Solving environment: ...working... done
2022-12-01T10:20:18.4485358Z 
2022-12-01T10:20:18.4485818Z ## Package Plan ##
2022-12-01T10:20:18.4486013Z 
2022-12-01T10:20:18.4486183Z   environment location: /opt/conda
2022-12-01T10:20:18.4486394Z 
2022-12-01T10:20:18.4486532Z   added / updated specs:
2022-12-01T10:20:18.4486922Z     - cmake
2022-12-01T10:20:18.4487075Z 
2022-12-01T10:20:18.4487095Z 
2022-12-01T10:20:18.4487252Z The following packages will be downloaded:
2022-12-01T10:20:18.4488327Z 
2022-12-01T10:20:18.4488494Z     package                    |            build
2022-12-01T10:20:18.4488892Z     ---------------------------|-----------------
2022-12-01T10:20:18.4489309Z     c-ares-1.18.1              |       h7f8727e_0         114 KB
2022-12-01T10:20:18.4489737Z     ca-certificates-2022.10.11 |       h06a4308_0         124 KB
2022-12-01T10:20:18.4490175Z     certifi-2022.9.24          |  py310h06a4308_0         154 KB
2022-12-01T10:20:18.4490570Z     cmake-3.22.1               |       h1fce559_0         7.3 MB
2022-12-01T10:20:18.4490958Z     expat-2.4.9                |       h6a678d5_0         156 KB
2022-12-01T10:20:18.4491350Z     krb5-1.19.2                |       hac12032_0         1.2 MB
2022-12-01T10:20:18.4492004Z     libcurl-7.84.0             |       h91b91d3_0         337 KB
2022-12-01T10:20:18.4492420Z     libedit-3.1.20210910       |       h7f8727e_0         166 KB
2022-12-01T10:20:18.4492810Z     libev-4.33                 |       h7f8727e_1         111 KB
2022-12-01T10:20:18.4493197Z     libgcc-ng-11.2.0           |       h1234567_1         5.3 MB
2022-12-01T10:20:18.4493571Z     libgomp-11.2.0             |       h1234567_1         474 KB
2022-12-01T10:20:18.4493970Z     libnghttp2-1.46.0          |       hce63b2e_0         680 KB
2022-12-01T10:20:18.4494374Z     libssh2-1.10.0             |       h8f2d780_0         274 KB
2022-12-01T10:20:18.4494764Z     libstdcxx-ng-11.2.0        |       h1234567_1         4.7 MB
2022-12-01T10:20:18.4495163Z     libuv-1.40.0               |       h7b6447c_0         736 KB
2022-12-01T10:20:18.4495562Z     lz4-c-1.9.3                |       h295c915_1         185 KB
2022-12-01T10:20:18.4495989Z     openssl-1.1.1s             |       h7f8727e_0         3.6 MB
2022-12-01T10:20:18.4496727Z     rhash-1.4.1                |       h3c74f83_1         203 KB
2022-12-01T10:20:18.4497393Z     zstd-1.5.2                 |       ha4553b6_0         488 KB
2022-12-01T10:20:18.4498231Z     ------------------------------------------------------------
2022-12-01T10:20:18.4498896Z                                            Total:        26.3 MB
2022-12-01T10:20:18.4499112Z 
2022-12-01T10:20:18.4499279Z The following NEW packages will be INSTALLED:
2022-12-01T10:20:18.4499482Z 
2022-12-01T10:20:18.4499879Z   c-ares             pkgs/main/linux-64::c-ares-1.18.1-h7f8727e_0 None
2022-12-01T10:20:18.4500389Z   cmake              pkgs/main/linux-64::cmake-3.22.1-h1fce559_0 None
2022-12-01T10:20:18.4500889Z   expat              pkgs/main/linux-64::expat-2.4.9-h6a678d5_0 None
2022-12-01T10:20:18.4501366Z   krb5               pkgs/main/linux-64::krb5-1.19.2-hac12032_0 None
2022-12-01T10:20:18.4501861Z   libcurl            pkgs/main/linux-64::libcurl-7.84.0-h91b91d3_0 None
2022-12-01T10:20:18.4502570Z   libedit            pkgs/main/linux-64::libedit-3.1.20210910-h7f8727e_0 None
2022-12-01T10:20:18.4503059Z   libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1 None
2022-12-01T10:20:18.4503576Z   libnghttp2         pkgs/main/linux-64::libnghttp2-1.46.0-hce63b2e_0 None
2022-12-01T10:20:18.4504107Z   libssh2            pkgs/main/linux-64::libssh2-1.10.0-h8f2d780_0 None
2022-12-01T10:20:18.4504605Z   libuv              pkgs/main/linux-64::libuv-1.40.0-h7b6447c_0 None
2022-12-01T10:20:18.4505084Z   lz4-c              pkgs/main/linux-64::lz4-c-1.9.3-h295c915_1 None
2022-12-01T10:20:18.4505576Z   rhash              pkgs/main/linux-64::rhash-1.4.1-h3c74f83_1 None
2022-12-01T10:20:18.4506066Z   zstd               pkgs/main/linux-64::zstd-1.5.2-ha4553b6_0 None
2022-12-01T10:20:18.4506280Z 
2022-12-01T10:20:18.4506481Z The following packages will be UPDATED:
2022-12-01T10:20:18.4506660Z 
2022-12-01T10:20:18.4506944Z   ca-certificates                     2022.07.19-h06a4308_0 --> 2022.10.11-h06a4308_0 None
2022-12-01T10:20:18.4507449Z   certifi                         2022.9.14-py310h06a4308_0 --> 2022.9.24-py310h06a4308_0 None
2022-12-01T10:20:18.4507911Z   libgcc-ng                               9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:20:18.4508346Z   libgomp                                 9.3.0-h5101ec6_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:20:18.4508802Z   libstdcxx-ng                            9.3.0-hd4cf53a_17 --> 11.2.0-h1234567_1 None
2022-12-01T10:20:18.4509264Z   openssl                                 1.1.1q-h7f8727e_0 --> 1.1.1s-h7f8727e_0 None
2022-12-01T10:20:18.4510030Z 
2022-12-01T10:20:18.4510050Z 
2022-12-01T10:20:19.8874052Z Preparing transaction: ...working... done
2022-12-01T10:20:20.5195056Z Verifying transaction: ...working... done
2022-12-01T10:20:21.4479592Z Executing transaction: ...working... done
2022-12-01T10:20:21.5627924Z Retrieving notices: ...working... done
2022-12-01T10:20:21.7867286Z ++ [[ linux-bionic-cuda11.6-py3.10-gcc7 == *centos* ]]
2022-12-01T10:20:21.7867738Z + echo 'Environment variables'
2022-12-01T10:20:21.7868264Z Environment variables
2022-12-01T10:20:21.7868531Z + env
2022-12-01T10:20:21.7876668Z SHARD_NUMBER=1
2022-12-01T10:20:21.7877194Z NV_LIBCUBLAS_DEV_VERSION=11.9.2.110-1
2022-12-01T10:20:21.7877952Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-6
2022-12-01T10:20:21.7878748Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2022-12-01T10:20:21.7879593Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.12.10-1+cuda11.6
2022-12-01T10:20:21.7880210Z UCC_HOME=/usr
2022-12-01T10:20:21.7880971Z BUILD_ENVIRONMENT=linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T10:20:21.7881820Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-6=11.6.3.124-1
2022-12-01T10:20:21.7882349Z INSTALLED_DB=yes
2022-12-01T10:20:21.7882884Z HOSTNAME=7ff5451e42d4
2022-12-01T10:20:21.7883384Z GITHUB_REF_NAME=89997/merge
2022-12-01T10:20:21.7884029Z GITHUB_API_URL=https://api.github.com
2022-12-01T10:20:21.7884610Z OPENSSL_DIR=/opt/openssl
2022-12-01T10:20:21.7885198Z UCC_COMMIT=12944da33f911daf505d9bbc51411233d0ed85e1
2022-12-01T10:20:21.7886699Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_fb52ea66-f825-44cf-85ee-225f1827d41f
2022-12-01T10:20:21.7887523Z CUDA_PATH=/usr/local/cuda
2022-12-01T10:20:21.7888554Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2022-12-01T10:20:21.7889255Z GITHUB_RUN_ATTEMPT=1
2022-12-01T10:20:21.7889746Z TEST_CONFIG=distributed
2022-12-01T10:20:21.7890347Z NV_LIBNPP_VERSION=11.6.3.124-1
2022-12-01T10:20:21.7891118Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-6=11.6.124-1
2022-12-01T10:20:21.7891716Z GITHUB_REPOSITORY_OWNER=pytorch
2022-12-01T10:20:21.7892280Z GITHUB_ACTIONS=true
2022-12-01T10:20:21.7892822Z NVIDIA_VISIBLE_DEVICES=all
2022-12-01T10:20:21.7893416Z NV_NVPROF_VERSION=11.6.124-1
2022-12-01T10:20:21.7894068Z NV_LIBCUSPARSE_VERSION=11.7.2.124-1
2022-12-01T10:20:21.7894598Z CI=true
2022-12-01T10:20:21.7895392Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2022-12-01T10:20:21.7896227Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-6=11.9.2.110-1
2022-12-01T10:20:21.7896813Z BRANCH=pull/89997
2022-12-01T10:20:21.7897461Z GITHUB_HEAD_REF=release/1.13-callstack
2022-12-01T10:20:21.7898151Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2022-12-01T10:20:21.7898885Z GITHUB_ACTOR=charlie-wt
2022-12-01T10:20:21.7899469Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2022-12-01T10:20:21.7900032Z GITHUB_ACTION_REF=
2022-12-01T10:20:21.7900559Z NCCL_VERSION=2.12.10-1
2022-12-01T10:20:21.7901013Z GITHUB_ACTION=__self
2022-12-01T10:20:21.7901491Z VALGRIND=ON
2022-12-01T10:20:21.7902022Z GITHUB_REF_PROTECTED=false
2022-12-01T10:20:21.7902907Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2022-12-01T10:20:21.7904031Z ***
2022-12-01T10:20:21.7904515Z INSTALLED_VISION=yes
2022-12-01T10:20:21.7904982Z NVARCH=x86_64
2022-12-01T10:20:21.7905599Z NV_LIBCUSPARSE_DEV_VERSION=11.7.2.124-1
2022-12-01T10:20:21.7906133Z HOME=/var/lib/jenkins
2022-12-01T10:20:21.7907201Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_fb52ea66-f825-44cf-85ee-225f1827d41f
2022-12-01T10:20:21.7908011Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2022-12-01T10:20:21.7908588Z GITHUB_ACTION_REPOSITORY=
2022-12-01T10:20:21.7909130Z GITHUB_REF_TYPE=branch
2022-12-01T10:20:21.7910159Z NV_LIBNCCL_PACKAGE_VERSION=2.12.10-1
2022-12-01T10:20:21.7910723Z GITHUB_RETENTION_DAYS=90
2022-12-01T10:20:21.7911530Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2022-12-01T10:20:21.7912350Z NV_LIBNCCL_PACKAGE=libnccl2=2.12.10-1+cuda11.6
2022-12-01T10:20:21.7913460Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_fb52ea66-f825-44cf-85ee-225f1827d41f
2022-12-01T10:20:21.7914288Z DEBIAN_FRONTEND=noninteractive
2022-12-01T10:20:21.7915017Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2022-12-01T10:20:21.7915644Z GITHUB_REF=refs/pull/89997/merge
2022-12-01T10:20:21.7916252Z NV_CUDA_LIB_VERSION=11.6.2-1
2022-12-01T10:20:21.7916860Z GITHUB_SHA=cdc064133ad5e5a46a756ee9218659e8f252e950
2022-12-01T10:20:21.7917648Z INSTALLED_PROTOBUF=yes
2022-12-01T10:20:21.7918204Z GITHUB_RUN_ID=3591403534
2022-12-01T10:20:21.7918927Z NV_LIBNPP_PACKAGE=libnpp-11-6=11.6.3.124-1
2022-12-01T10:20:21.7919525Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2022-12-01T10:20:21.7920118Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2022-12-01T10:20:21.7920943Z NV_NVTX_VERSION=11.6.124-1
2022-12-01T10:20:21.7921521Z GITHUB_SERVER_URL=https://github.com
2022-12-01T10:20:21.7922068Z MAX_JOBS=30
2022-12-01T10:20:21.7922636Z NV_LIBCUBLAS_VERSION=11.9.2.110-1
2022-12-01T10:20:21.7923356Z NV_LIBCUBLAS_PACKAGE=libcublas-11-6=11.9.2.110-1
2022-12-01T10:20:21.7924345Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2022-12-01T10:20:21.7925039Z UCX_HOME=/usr
2022-12-01T10:20:21.7925528Z PYTORCH_RETRY_TEST_CASES=1
2022-12-01T10:20:21.7926173Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2022-12-01T10:20:21.7926879Z BASE_SHA=ae2fe4033cf3b17259b17f351020b988fa893f91
2022-12-01T10:20:21.7927569Z NV_CUDA_CUDART_DEV_VERSION=11.6.55-1
2022-12-01T10:20:21.7930097Z PR_BODY=Link to landed master PR (if applicable):https://github.com/pytorch/pytorch/pull/88993Criteria category:1: This prevents a crash, which was introduced [here](https://github.com/pytorch/pytorch/commit/4b7de265569f7fd731dd1cfea83ce804cc22f7c0#diff-45b117ca26b4fc9174fbe0d7a9cd8cb1c43964cd5e2bb20c7778ee00a942ef63), tagged for 1.132: Prevents a crashIm hoping this is a low-risk change, since its just changing one method for its safer form.
2022-12-01T10:20:21.7931903Z GITHUB_BASE_REF=release/1.13
2022-12-01T10:20:21.7932395Z TERM=xterm
2022-12-01T10:20:21.7932834Z XLA_CUDA=
2022-12-01T10:20:21.7933386Z NV_NVML_DEV_VERSION=11.6.55-1
2022-12-01T10:20:21.7933931Z TORCH_CUDA_ARCH_LIST=Maxwell
2022-12-01T10:20:21.7934458Z CUDA_VERSION=11.6.2
2022-12-01T10:20:21.7935153Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-6
2022-12-01T10:20:21.7935753Z OPENSSL_ROOT_DIR=/opt/openssl
2022-12-01T10:20:21.7937033Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_fb52ea66-f825-44cf-85ee-225f1827d41f
2022-12-01T10:20:21.7937829Z GITHUB_JOB=test
2022-12-01T10:20:21.7938318Z SCCACHE_S3_KEY_PREFIX=pull
2022-12-01T10:20:21.7967910Z COMMIT_MESSAGES=+ 56508e29e6b1847e050462d818021a0389d0de99 Release 1.13, Install torch from test channel, Pin build… (#86290)- f682048bf9475d324044e955d8d67464ee446c3a Fix for the binary upload (#86385)+ 95112ca043c37e38101e629c1f69663e911cc909 Fix binary builds for the release - unblock release (#86484)- c38dbd0e1dd88cce783c8dce6cce1a97276b6bb9 Conditionally build the TestApp benchmark based on lite interpreter (#86314) (#86377)- 50a9cd95baf29e136fdf5f027c6f4c4a7ccd5b9b Add version selector back to functorch docs (#86602) (#86689)- 5be45fc4f1a86624fe5ce0e99a2a5558011ca69e [ROCm] set nvfuser default to disabled, keep CI (#86369) (#86725)- f37023b03f3e92dadb247e89fd4e024eb4a0eb8a [MPS] Better error message for `slow_conv2d_forward` (#86844)- 786431cd13d171965feeb17240cc7bf8dc509dba [DOC] Use type hints to show annotation in the docs  (#86851)- 03992a6fb3dbd371fe41d3cd95c6589b45a10f14 Make the data types of output and input consistenst for batchnorm (#86784)+ d9ddab5efa2224704e4b982e651589b31b1c1547 [1.13] Remove torch.vmap (#86333)+ 01042487e2d848d17351555ea73c61d38a60a7f9 [1.13] Release-only improvements to functorch docs (#86693)- 71251e2521ac4b737af43d536dff359df256eadd ci: Just use regular checkout (#86824) (#86895)+ f65ac2ad20f8eab95a0d0fb93c3550afc2b86d91 [CI] Fix builder ref for release, linux only (#86904)- 51c16f092ca8d7802decfd56d34ae99d1c0f335c [functorch] Add more details to the functorch install page (#86823) (#86903)- 311b47b72c73509a82ed0b613a7c563e6628332b [quant] Move the order of x86 engine to avoid changing the default qengine (#86631) (#86726)+ 7e196948ca0c5ede7fa6e0831fda18135e6573a5 Add vmap support for slogdet; fix regression from functorch 0.2.1 (#86815) (#86902)- aab841e89d850c9da9048a64541cfa78b222f1ff [ONNX] Support device().type() string comparison with constant (#86168) (#86921)- de3fa48e38e5370d18748e131bd20db53390ea5c Install c10d headers with absolute path (#86257) (#86933)+ 366c59e2606368a6f2ee7165d58686d0d494dd92 Allow PrivateUse1 backends to not have Storage (#86557) (#86803)+ 492d572086a93a31fc8972d29726453ee1e46717 [functorch] fix cross (#86934)- f89a762a8efcca77722071ec042a02742ec4537d Fix the performance issue that the for-loop before ExternallCall could not be parallelized. (#85056) (#86516)- c40a04415c59edc45b2ceba682377b255f2091c7 Add error checking to flaky test bot platform parser (#86632) (#87201)+ eda8263f151a17d0d81d33e3f48fb2102daf4c14 [Release 1.13.0][DataPipe] Documentation and interface fix (#87140)+ d2325f3b6a17e54b5d48c68d82b74354dfd35f61 handle libomp update on circleci (#86979) (#87132)- 7688e5eeabce25b67aa99280772d094b1c5f7091 [ONNX] Fix triu/tril export with diagonal input (#86843) (#86925)- 1ebff1dcaf75d0305dc713eea104ddfa3ae6b588 [ONNX] Renable assert diagnostic test (#85999) (#86924)- 1442f04f3bab3b0e0f22fbb68c8f328e17801c69 Enables lazy loading of cuda modules if not set by user (#86509)+ d324341922beb064057e12daae19199480885d5e Bug fix for torch._C._willEngineExecuteNode (#86672)- 894bad72c9dfb946e9bdfebaf2db41473b3dd4fe [ONNX] Fix scalar_type_analysis metadata for copied constant (#86716) (#86923)- ef6dd0d34d379604eec8db34dd58581b8b061249 [ONNX] Ignore print(Tensor) during tracing (#86223) (#87122)- e6d8c1392c817636c15ad974dd160292cdc471bc [einsum] Fix opt_einsum defaults to be more reasonable (#86985) (#87144)+ 3af7c7fc98af441ca90ef38b262497b530a317fb [einsum] fix MPS regression and fix incorrect contraction order when path is None (#87261)- 9e3df4920f75fa2aaf5bbe04ed302fc80008a8b9 Improve NestedTensor documentation (#85186) (#87337)- 3ca59f4b8aff32fbad3fcd13b812b5335047b8ce Reenable aot tests on windows for cuda 11.7 and up (#87193) (#87307)- 0948dbb3e7d6e5e99305b71423f7e9a45a6e5e14 Add torch.sparse overview section (#85265) (#87376)- 1f80ac7fa72cacafbb4901109b465db05f91d818 [Docs] Update mm family ops and F.linear to note limited sparse support.  (#86220) (#87379)- 7342a3b0eac2e3ebf4d9b5ce1c3c73c4134c8751 Add prototype warning to MaskedTensor constructor (#87107) (#87380)- aba5544affda233f8413b5fd74df1cdf7ed91c18 [maskedtensor] add docs (#84887) (#87381)- 6e2cb22d74d1939e0aa5c00c665a58e8796bef31 [functorch][docs] Downgrade the warning about forward-mode AD coverage (#87383) (#87386)+ ca0ee922bc9440bfc86b332d69ee0559f748bc6e [release] Add warning to stateless.functional_call for deprecated behavior (#87079)- d2ce4310bad94f2e30fd75cb14e175bcef7ba6fa Assert if padding mask type is unexpected (#87106) (#87384)- 05cb998e11d109a55ddee0fb72c17aa07fa8104a [MPS] Do not dispatch empty job in  `bitwise_not` (#87286) (#87427)- 385022c76e4229a9393d63871225e0961e91c2db [ci] handle libomp upgrade on github (#87382) (#87408)- 3ffddc0b2d5fc963463c98b518b0eb1f88788a74 [functorch] fix AOTAutograd tutorial (#87415) (#87434)+ 55c76baf579cb6593f87d1a23e9a49afeb55f15a [1.13] Fix functorch version docs (#87392)- 59686b4c60289188436ac61576fd389821858fa5 [maskedtensor] fix docs formatting (#87387) (#87406)+ 0c0df0be7497112022804df03aeeb0fcbadc9243 Add `weights_only` option to `torch.load` (#87443)+ d253eb29d86af51ae17b950825ffdb5661b5af7f Avoid calling logging.basicConfig (#86959) (#87455)- d3aecbd9bc58a42366abffa63748d73c872bd927 Delete torch::deploy from pytorch core (#85953) (#85953) (#87454)- 51fa4fae41367a17f49ce648c7cdd6aa72f6e6ac Move PadNd from ATen/native to ATen (#87456)- f6c42ae2c29ba788523149dbcc791bf14530f93d Reenable `isinstance` with `torch.distributed.ReduceOp` (#87303) (#87463)- 6a8be2cb630ce79d654a707e5ea454d013acbda1 [ONNX] Reland: Update training state logic to support ScriptedModule (#86745) (#87457)+ 8569a44f38ff906103f40c6925dec366c7557943 [MPS] Revamp copy_to_mps_ implementation (#87475)- fdb18da4cc3ee04c2423eb33b2780edc4cae64e0 Fix distributed issue by including distributed files (#87612)- 341c377c0f595ff4e4a0fbdef21978c038b64b98 Add General Project Policies (#87385) (#87613)- 4e1a4b150a4ccd5f401a4293a0336a9a8a1dad2d fix docs push (#87498) (#87628)- 7c98e70d44abc7a1aead68b6ea6c8adc8c554db5 attempted fix for nvrtc with lovelace (#87611) (#87618)- 74a9ca993bd79f8131829e9c946657fa9a1d05ef [JIT][Security] Do not blindly eval input string (#89189) (#89925)+ ae2fe4033cf3b17259b17f351020b988fa893f91 Update masked.rst (#89758) (#89923)- c13d400bffe90e16b96520bbc8a41a6f0c9cd584 Use the Python frame safely in _pythonCallstack
2022-12-01T10:20:21.7982135Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2022-12-01T10:20:21.7982731Z NUM_TEST_SHARDS=3
2022-12-01T10:20:21.7983206Z PR_NUMBER=89997
2022-12-01T10:20:21.7984346Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_fb52ea66-f825-44cf-85ee-225f1827d41f
2022-12-01T10:20:21.7985110Z SHLVL=1
2022-12-01T10:20:21.7985787Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-6
2022-12-01T10:20:21.7986447Z GITHUB_REPOSITORY=pytorch/pytorch
2022-12-01T10:20:21.7987612Z NVIDIA_REQUIRE_CUDA=cuda>=11.6 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471
2022-12-01T10:20:21.7988912Z NV_LIBNPP_DEV_VERSION=11.6.3.124-1
2022-12-01T10:20:21.7990052Z SHA1=c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T10:20:21.7990674Z GITHUB_EVENT_NAME=pull_request
2022-12-01T10:20:21.7991335Z NV_CUDA_CUDART_VERSION=11.6.55-1
2022-12-01T10:20:21.7992032Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2022-12-01T10:20:21.7992621Z GITHUB_RUN_NUMBER=69319
2022-12-01T10:20:21.7993147Z GITHUB_WORKFLOW=pull
2022-12-01T10:20:21.7994080Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:20:21.7994997Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.12.10-1
2022-12-01T10:20:21.7995898Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T10:20:21.7996719Z GITHUB_TRIGGERING_ACTOR=charlie-wt
2022-12-01T10:20:21.7997471Z _=/usr/bin/env
2022-12-01T10:20:21.7998063Z + echo 'Testing pytorch'
2022-12-01T10:20:21.7998573Z Testing pytorch
2022-12-01T10:20:21.7999138Z + export LANG=C.UTF-8
2022-12-01T10:20:21.7999674Z + LANG=C.UTF-8
2022-12-01T10:20:21.8000132Z + PR_NUMBER=89997
2022-12-01T10:20:21.8000650Z + [[ distributed == \d\e\f\a\u\l\t ]]
2022-12-01T10:20:21.8001250Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2022-12-01T10:20:21.8002049Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-12-01T10:20:21.8002683Z + [[ distributed == \s\l\o\w ]]
2022-12-01T10:20:21.8003535Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *slow-gradcheck* ]]
2022-12-01T10:20:21.8004437Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-12-01T10:20:21.8005113Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-12-01T10:20:21.8005783Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2022-12-01T10:20:21.8006619Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda11* ]]
2022-12-01T10:20:21.8007242Z + export BUILD_SPLIT_CUDA=ON
2022-12-01T10:20:21.8007792Z + BUILD_SPLIT_CUDA=ON
2022-12-01T10:20:21.8008335Z + [[ distributed == *crossref* ]]
2022-12-01T10:20:21.8008867Z + [[ distributed == *dynamo* ]]
2022-12-01T10:20:21.8009465Z + [[ -n 89997 ]]
2022-12-01T10:20:21.8009972Z + [[ -z '' ]]
2022-12-01T10:20:21.8010503Z + export PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-12-01T10:20:21.8011152Z + PYTORCH_TEST_SKIP_CUDA_MEM_LEAK_CHECK=1
2022-12-01T10:20:21.8012018Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *rocm* ]]
2022-12-01T10:20:21.8012879Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *-bazel-* ]]
2022-12-01T10:20:21.8013614Z + pip_install --user ninja
2022-12-01T10:20:21.8014357Z + pip install --progress-bar off --user ninja
2022-12-01T10:20:22.3795586Z Collecting ninja
2022-12-01T10:20:22.4015711Z   Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
2022-12-01T10:20:23.2896834Z Installing collected packages: ninja
2022-12-01T10:20:23.3010932Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2022-12-01T10:20:23.3011818Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2022-12-01T10:20:23.3068604Z Successfully installed ninja-1.11.1
2022-12-01T10:20:23.3766190Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:20:23.3766824Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2022-12-01T10:20:23.3767732Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *asan* ]]
2022-12-01T10:20:23.3768098Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2022-12-01T10:20:23.3768422Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2022-12-01T10:20:23.3772429Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *tbb* ]]
2022-12-01T10:20:23.3787264Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-12-01T10:20:23.3787733Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *-bazel-* ]]
2022-12-01T10:20:23.3792370Z + cd test
2022-12-01T10:20:23.3793754Z + python -c 'import torch; print(torch.__config__.show())'
2022-12-01T10:20:24.6300620Z PyTorch built with:
2022-12-01T10:20:24.6301118Z   - GCC 7.5
2022-12-01T10:20:24.6301425Z   - C++ Version: 201402
2022-12-01T10:20:24.6301948Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-12-01T10:20:24.6302531Z   - Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-12-01T10:20:24.6302951Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-12-01T10:20:24.6303314Z   - LAPACK is enabled (usually provided by MKL)
2022-12-01T10:20:24.6303651Z   - NNPACK is enabled
2022-12-01T10:20:24.6303970Z   - CPU capability usage: AVX2
2022-12-01T10:20:24.6304293Z   - CUDA Runtime 11.6
2022-12-01T10:20:24.6304677Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2022-12-01T10:20:24.6305407Z   - CuDNN 8.3.2  (built against CUDA 11.5)
2022-12-01T10:20:24.6305710Z   - Magma 2.6.1
2022-12-01T10:20:24.6308712Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.6, CUDNN_VERSION=8.3.2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.13.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2022-12-01T10:20:24.6311480Z 
2022-12-01T10:20:24.8157156Z + cd test
2022-12-01T10:20:24.8157750Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2022-12-01T10:20:26.0287715Z ATen/Parallel:
2022-12-01T10:20:26.0288317Z 	at::get_num_threads() : 16
2022-12-01T10:20:26.0288702Z 	at::get_num_interop_threads() : 16
2022-12-01T10:20:26.0289010Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2022-12-01T10:20:26.0289291Z 	omp_get_max_threads() : 16
2022-12-01T10:20:26.0289939Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2022-12-01T10:20:26.0290349Z 	mkl_get_max_threads() : 16
2022-12-01T10:20:26.0291073Z Intel(R) MKL-DNN v2.6.0 (Git Hash 52b5f107dd9cf10910aaa19cb47f3abf9b349815)
2022-12-01T10:20:26.0291478Z std::thread::hardware_concurrency() : 32
2022-12-01T10:20:26.0291754Z Environment variables:
2022-12-01T10:20:26.0292038Z 	OMP_NUM_THREADS : [not set]
2022-12-01T10:20:26.0292310Z 	MKL_NUM_THREADS : [not set]
2022-12-01T10:20:26.0292571Z ATen parallel backend: OpenMP
2022-12-01T10:20:26.0292750Z 
2022-12-01T10:20:26.2033283Z + [[ distributed == *backward* ]]
2022-12-01T10:20:26.2033624Z + [[ distributed == *xla* ]]
2022-12-01T10:20:26.2033901Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2022-12-01T10:20:26.2034452Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *libtorch* ]]
2022-12-01T10:20:26.2034790Z + [[ distributed == distributed ]]
2022-12-01T10:20:26.2035048Z + install_torchdynamo
2022-12-01T10:20:26.2035297Z + local commit
2022-12-01T10:20:26.2038011Z ++ get_pinned_commit torchdynamo
2022-12-01T10:20:26.2038340Z ++ cat .github/ci_commit_pins/torchdynamo.txt
2022-12-01T10:20:26.2053829Z + commit=6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:26.2054466Z + pip_install --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:26.2055196Z + pip install --progress-bar off --user git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:26.6998301Z Collecting git+https://github.com/pytorch/torchdynamo.git@6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:26.7004252Z   Cloning https://github.com/pytorch/torchdynamo.git (to revision 6ead5cae0d1234aa64db06fe230ef56e12ec76fe) to /tmp/pip-req-build-w114sndh
2022-12-01T10:20:26.7024330Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchdynamo.git /tmp/pip-req-build-w114sndh
2022-12-01T10:20:27.3153281Z   Running command git rev-parse -q --verify 'sha^6ead5cae0d1234aa64db06fe230ef56e12ec76fe'
2022-12-01T10:20:27.3175480Z   Running command git fetch -q https://github.com/pytorch/torchdynamo.git 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:27.5992347Z   Running command git checkout -q 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:28.1236051Z   Resolved https://github.com/pytorch/torchdynamo.git to commit 6ead5cae0d1234aa64db06fe230ef56e12ec76fe
2022-12-01T10:20:30.1136529Z   Preparing metadata (setup.py) ... [?25l- done
2022-12-01T10:20:30.1205456Z [?25hRequirement already satisfied: torch>=1.12.0 in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.13.0a0+gitc13d400)
2022-12-01T10:20:30.1210045Z Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.21.2)
2022-12-01T10:20:30.1639840Z Collecting tabulate
2022-12-01T10:20:30.1808199Z   Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)
2022-12-01T10:20:30.1879444Z Requirement already satisfied: pyyaml in /opt/conda/lib/python3.10/site-packages/PyYAML-6.0-py3.10-linux-x86_64.egg (from torchdynamo==1.13.0.dev0) (6.0)
2022-12-01T10:20:30.1884294Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torchdynamo==1.13.0.dev0) (1.11.1)
2022-12-01T10:20:30.1923684Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch>=1.12.0->torchdynamo==1.13.0.dev0) (4.3.0)
2022-12-01T10:20:30.1959874Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torchdynamo==1.13.0.dev0) (1.2.1)
2022-12-01T10:20:30.2084991Z Building wheels for collected packages: torchdynamo
2022-12-01T10:20:34.5379444Z   Building wheel for torchdynamo (setup.py) ... [?25l- \ | / - done
2022-12-01T10:20:34.5481701Z [?25h  Created wheel for torchdynamo: filename=torchdynamo-1.13.0.dev0-cp310-cp310-linux_x86_64.whl size=2693850 sha256=fa304b5d8404f4318372429e5b67b4c26c7deb2a2a1d5bccc961b74826593424
2022-12-01T10:20:34.5483106Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/6c/b6/c8/fbfe87a581aa24ccebb62be69424794eca3c035823c5b100d9
2022-12-01T10:20:34.5506353Z Successfully built torchdynamo
2022-12-01T10:20:35.4282975Z Installing collected packages: tabulate, torchdynamo
2022-12-01T10:20:35.7989004Z Successfully installed tabulate-0.9.0 torchdynamo-1.13.0.dev0
2022-12-01T10:20:35.8908954Z + test_distributed
2022-12-01T10:20:35.8909782Z + echo 'Testing distributed python tests'
2022-12-01T10:20:35.8910128Z Testing distributed python tests
2022-12-01T10:20:35.8910582Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose
2022-12-01T10:20:37.6499653Z Ignoring disabled issues:  []
2022-12-01T10:20:37.6881167Z /var/lib/jenkins/workspace/test/run_test.py:1050: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2022-12-01T10:20:37.6882287Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2022-12-01T10:20:37.6886087Z Found test time stats from artifacts
2022-12-01T10:20:37.6897948Z Selected tests:
2022-12-01T10:20:37.6898695Z  distributed/test_distributed_spawn
2022-12-01T10:20:37.6899399Z  distributed/_shard/checkpoint/test_file_system_checkpoint
2022-12-01T10:20:37.6899855Z  distributed/_shard/checkpoint/test_utils
2022-12-01T10:20:37.6900150Z  distributed/_shard/test_sharder
2022-12-01T10:20:37.6900474Z  distributed/fsdp/test_flatten_params_wrapper
2022-12-01T10:20:37.6900809Z  distributed/pipeline/sync/skip/test_api
2022-12-01T10:20:37.6901116Z  distributed/pipeline/sync/skip/test_leak
2022-12-01T10:20:37.6901441Z  distributed/pipeline/sync/skip/test_tracker
2022-12-01T10:20:37.6901761Z  distributed/pipeline/sync/test_bugs
2022-12-01T10:20:37.6902078Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-12-01T10:20:37.6902423Z  distributed/pipeline/sync/test_microbatch
2022-12-01T10:20:37.6902744Z  distributed/pipeline/sync/test_pipeline
2022-12-01T10:20:37.6903047Z  distributed/pipeline/sync/test_worker
2022-12-01T10:20:37.8848141Z Prioritized test from test file changes.
2022-12-01T10:20:37.8848871Z reordering tests for PR:
2022-12-01T10:20:37.8849630Z prioritized: ['distributed/_shard/checkpoint/test_file_system_checkpoint', 'distributed/_shard/checkpoint/test_utils', 'distributed/fsdp/test_flatten_params_wrapper']
2022-12-01T10:20:37.8850946Z the rest: ['distributed/test_distributed_spawn', 'distributed/_shard/test_sharder', 'distributed/pipeline/sync/skip/test_api', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_worker']
2022-12-01T10:20:37.8851650Z 
2022-12-01T10:20:37.8852167Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2022-12-01T10:20:37.9061248Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2022-12-01T10:20:37.9244844Z parallel (file granularity) tests:
2022-12-01T10:20:37.9245358Z  
2022-12-01T10:20:37.9245793Z serial (file granularity) tests:
2022-12-01T10:20:37.9246341Z  distributed/_shard/checkpoint/test_file_system_checkpoint
2022-12-01T10:20:37.9246697Z  distributed/_shard/checkpoint/test_utils
2022-12-01T10:20:37.9247030Z  distributed/fsdp/test_flatten_params_wrapper
2022-12-01T10:20:37.9247333Z  distributed/test_distributed_spawn
2022-12-01T10:20:37.9247634Z  distributed/_shard/test_sharder
2022-12-01T10:20:37.9247951Z  distributed/pipeline/sync/skip/test_api
2022-12-01T10:20:37.9248262Z  distributed/pipeline/sync/skip/test_leak
2022-12-01T10:20:37.9248588Z  distributed/pipeline/sync/skip/test_tracker
2022-12-01T10:20:37.9248909Z  distributed/pipeline/sync/test_bugs
2022-12-01T10:20:37.9249227Z  distributed/pipeline/sync/test_deferred_batch_norm
2022-12-01T10:20:37.9249581Z  distributed/pipeline/sync/test_microbatch
2022-12-01T10:20:37.9250072Z  distributed/pipeline/sync/test_pipeline
2022-12-01T10:20:37.9250393Z  distributed/pipeline/sync/test_worker
2022-12-01T10:20:39.6885738Z Ignoring disabled issues:  []
2022-12-01T10:20:39.6980948Z Ignoring disabled issues:  []
2022-12-01T10:20:40.0118729Z Running distributed/_shard/checkpoint/test_file_system_checkpoint ... [2022-12-01 10:20:40.011347]
2022-12-01T10:20:40.0121191Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:20:40.011780]
2022-12-01T10:21:02.1203775Z 
2022-12-01T10:21:02.1206714Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_file_system_checkpoint
2022-12-01T10:21:02.1211845Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_1rn95ew8)
2022-12-01T10:21:02.1212314Z 
2022-12-01T10:21:02.1212416Z Running tests...
2022-12-01T10:21:02.1212941Z ----------------------------------------------------------------------
2022-12-01T10:21:02.1213564Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint
2022-12-01T10:21:02.1214175Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:21:02.1214661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 737
2022-12-01T10:21:02.1215117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 738
2022-12-01T10:21:02.1215742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1216204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1216770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1217514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1218114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1218570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1219137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1219614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1220064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:21:02.1220524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:21:02.1221021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:21:02.1221529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:21:02.1222204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1222882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1223317Z ok (5.091s)
2022-12-01T10:21:02.1223797Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 820
2022-12-01T10:21:02.1224365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 821
2022-12-01T10:21:02.1224964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1225419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1226106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1226608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1227176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1227638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1228215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1228691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1229113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:21:02.1230192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:21:02.1230694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:21:02.1231180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:21:02.1231856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1232551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1232945Z ok (4.012s)
2022-12-01T10:21:02.1233379Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 903
2022-12-01T10:21:02.1233919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 904
2022-12-01T10:21:02.1234528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1235090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1235685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1236161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1236740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1237168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1237747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1238217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1238656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:21:02.1239111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:21:02.1239605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:21:02.1240102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:21:02.1240742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1241440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1241830Z ok (3.510s)
2022-12-01T10:21:02.1242312Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 986
2022-12-01T10:21:02.1242860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 987
2022-12-01T10:21:02.1243468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1244001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1244606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1245055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1245637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1246113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1246690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1247135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1247576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:21:02.1248073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:21:02.1248547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:21:02.1249034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:21:02.1249693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1250384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1250759Z ok (4.112s)
2022-12-01T10:21:02.1251127Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... ok (0.044s)
2022-12-01T10:21:02.1251776Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1069
2022-12-01T10:21:02.1252460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1070
2022-12-01T10:21:02.1253057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1253508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1254084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1254538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1255121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:21:02.1255567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:21:02.1256143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:21:02.1256591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:21:02.1257043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:21:02.1257532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:21:02.1258021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:21:02.1258482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:21:02.1259136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1259827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:21:02.1260199Z ok (3.510s)
2022-12-01T10:21:02.1260348Z 
2022-12-01T10:21:02.1260624Z ----------------------------------------------------------------------
2022-12-01T10:21:02.1260962Z Ran 6 tests in 20.281s
2022-12-01T10:21:02.1261128Z 
2022-12-01T10:21:02.1261276Z OK
2022-12-01T10:21:02.1261399Z 
2022-12-01T10:21:02.1261531Z Generating XML reports...
2022-12-01T10:21:02.1262230Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221201102041.xml
2022-12-01T10:21:02.1263164Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221201102041.xml
2022-12-01T10:21:02.1264196Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201102041.xml
2022-12-01T10:21:02.1264671Z 
2022-12-01T10:21:02.1265010Z ##[endgroup]
2022-12-01T10:21:02.1265717Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_file_system_checkpoint_1rn95ew8)
2022-12-01T10:21:02.1266146Z 
2022-12-01T10:21:02.1266439Z Running distributed/_shard/checkpoint/test_utils ... [2022-12-01 10:21:02.120506]
2022-12-01T10:21:02.1267131Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/checkpoint/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:21:02.120808]
2022-12-01T10:21:05.5084561Z 
2022-12-01T10:21:05.5085114Z Expand the folded group to see the log file of distributed/_shard/checkpoint/test_utils
2022-12-01T10:21:05.5086521Z ##[group]PRINTING LOG FILE of distributed/_shard/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_utils_wsr5morp)
2022-12-01T10:21:05.5086917Z 
2022-12-01T10:21:05.5087019Z Running tests...
2022-12-01T10:21:05.5087532Z ----------------------------------------------------------------------
2022-12-01T10:21:05.5088201Z Test results will be stored in test-reports/python-unittest/distributed._shard.checkpoint.test_utils
2022-12-01T10:21:05.5088899Z   test_flat_data (__main__.TestMedatadaIndex) ... ok (1.580s)
2022-12-01T10:21:05.5089281Z   test_index_hint_ignored_on_equals (__main__.TestMedatadaIndex) ... ok (0.001s)
2022-12-01T10:21:05.5089696Z   test_index_hint_ignored_on_hash (__main__.TestMedatadaIndex) ... ok (0.001s)
2022-12-01T10:21:05.5090101Z   test_init_convert_offset (__main__.TestMedatadaIndex) ... ok (0.001s)
2022-12-01T10:21:05.5090485Z   test_sharded_tensor_lookup (__main__.TestMedatadaIndex) ... ok (0.003s)
2022-12-01T10:21:05.5090723Z 
2022-12-01T10:21:05.5091003Z ----------------------------------------------------------------------
2022-12-01T10:21:05.5091335Z Ran 5 tests in 1.586s
2022-12-01T10:21:05.5091498Z 
2022-12-01T10:21:05.5091592Z OK
2022-12-01T10:21:05.5091709Z 
2022-12-01T10:21:05.5091834Z Generating XML reports...
2022-12-01T10:21:05.5092453Z Generated XML report: test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20221201102103.xml
2022-12-01T10:21:05.5092818Z 
2022-12-01T10:21:05.5093134Z ##[endgroup]
2022-12-01T10:21:05.5093750Z FINISHED PRINTING LOG FILE of distributed/_shard/checkpoint/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-checkpoint-test_utils_wsr5morp)
2022-12-01T10:21:05.5094126Z 
2022-12-01T10:21:05.5094436Z Running distributed/fsdp/test_flatten_params_wrapper ... [2022-12-01 10:21:05.508537]
2022-12-01T10:21:05.5095154Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_flatten_params_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:21:05.508882]
2022-12-01T10:21:09.8207241Z 
2022-12-01T10:21:09.8207728Z Expand the folded group to see the log file of distributed/fsdp/test_flatten_params_wrapper
2022-12-01T10:21:09.8208752Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_flatten_params_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_flatten_params_wrapper__s_4y3im)
2022-12-01T10:21:09.8209166Z 
2022-12-01T10:21:09.8209281Z Running tests...
2022-12-01T10:21:09.8210264Z ----------------------------------------------------------------------
2022-12-01T10:21:09.8210875Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper
2022-12-01T10:21:09.8211365Z   test_empty_module (__main__.TestFlattenParams) ... ok (1.580s)
2022-12-01T10:21:09.8211750Z   test_flatten_nothing (__main__.TestFlattenParams) ... ok (0.009s)
2022-12-01T10:21:09.8212129Z   test_num_params (__main__.TestFlattenParams) ... ok (0.010s)
2022-12-01T10:21:09.8212475Z   test_output (__main__.TestFlattenParams) ... ok (0.026s)
2022-12-01T10:21:09.8212856Z   test_partial_flattening (__main__.TestFlattenParams) ... ok (0.012s)
2022-12-01T10:21:09.8213248Z   test_sharded_flat_param (__main__.TestFlattenParams) ... ok (0.009s)
2022-12-01T10:21:09.8213635Z   test_shared_params_num_params (__main__.TestFlattenParams) ... ok (0.011s)
2022-12-01T10:21:09.8214045Z   test_shared_params_output (__main__.TestFlattenParams) ... ok (0.022s)
2022-12-01T10:21:09.8214472Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParams) ... ok (0.041s)
2022-12-01T10:21:09.8214886Z   test_empty_module (__main__.TestFlattenParamsCUDA) ... ok (0.002s)
2022-12-01T10:21:09.8215268Z   test_flatten_nothing (__main__.TestFlattenParamsCUDA) ... ok (0.010s)
2022-12-01T10:21:09.8215660Z   test_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.012s)
2022-12-01T10:21:09.8216040Z   test_output (__main__.TestFlattenParamsCUDA) ... ok (0.428s)
2022-12-01T10:21:09.8216417Z   test_partial_flattening (__main__.TestFlattenParamsCUDA) ... ok (0.015s)
2022-12-01T10:21:09.8216832Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDA) ... ok (0.008s)
2022-12-01T10:21:09.8217256Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDA) ... ok (0.012s)
2022-12-01T10:21:09.8217681Z   test_shared_params_output (__main__.TestFlattenParamsCUDA) ... ok (0.024s)
2022-12-01T10:21:09.8218098Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDA) ... ok (0.051s)
2022-12-01T10:21:09.8218661Z   test_empty_module (__main__.TestFlattenParamsCUDAHalf) ... ok (0.002s)
2022-12-01T10:21:09.8219085Z   test_flatten_nothing (__main__.TestFlattenParamsCUDAHalf) ... ok (0.012s)
2022-12-01T10:21:09.8219650Z   test_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.014s)
2022-12-01T10:21:09.8220145Z   test_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.026s)
2022-12-01T10:21:09.8220573Z   test_partial_flattening (__main__.TestFlattenParamsCUDAHalf) ... ok (0.026s)
2022-12-01T10:21:09.8221405Z   test_sharded_flat_param (__main__.TestFlattenParamsCUDAHalf) ... ok (0.008s)
2022-12-01T10:21:09.8221937Z   test_shared_params_num_params (__main__.TestFlattenParamsCUDAHalf) ... ok (0.014s)
2022-12-01T10:21:09.8222394Z   test_shared_params_output (__main__.TestFlattenParamsCUDAHalf) ... ok (0.026s)
2022-12-01T10:21:09.8222847Z   test_shared_params_pnorm_after_step (__main__.TestFlattenParamsCUDAHalf) ... ok (0.056s)
2022-12-01T10:21:09.8223121Z 
2022-12-01T10:21:09.8223402Z ----------------------------------------------------------------------
2022-12-01T10:21:09.8223736Z Ran 27 tests in 2.468s
2022-12-01T10:21:09.8223900Z 
2022-12-01T10:21:09.8223993Z OK
2022-12-01T10:21:09.8224127Z 
2022-12-01T10:21:09.8224250Z Generating XML reports...
2022-12-01T10:21:09.8224876Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20221201102106.xml
2022-12-01T10:21:09.8225699Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20221201102106.xml
2022-12-01T10:21:09.8226539Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20221201102106.xml
2022-12-01T10:21:09.8226924Z 
2022-12-01T10:21:09.8227228Z ##[endgroup]
2022-12-01T10:21:09.8228354Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_flatten_params_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_flatten_params_wrapper__s_4y3im)
2022-12-01T10:21:09.8229183Z 
2022-12-01T10:21:09.8230236Z Running distributed/test_distributed_spawn ... [2022-12-01 10:21:09.820802]
2022-12-01T10:21:09.8230779Z /usr/bin/mpiexec
2022-12-01T10:21:09.8231264Z MPI not available -- MPI backend tests will be skipped
2022-12-01T10:21:09.8387200Z Running distributed tests for the test backend with env init_method
2022-12-01T10:21:09.8412527Z Running distributed tests for the nccl backend with env init_method
2022-12-01T10:21:09.8425268Z Running distributed tests for the gloo backend with env init_method
2022-12-01T10:21:09.8426037Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:21:09.842190]
2022-12-01T10:21:09.8439154Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:21:09.843500]
2022-12-01T10:21:09.8439725Z Running distributed tests for the ucc backend with env init_method
2022-12-01T10:21:09.8452269Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:21:09.844812]
2022-12-01T10:21:09.8464437Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:21:09.846086]
2022-12-01T10:21:11.9350613Z 
2022-12-01T10:21:11.9351325Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:21:11.9352307Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_x867oa81)
2022-12-01T10:21:11.9352802Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:21:11.9353295Z 
2022-12-01T10:21:11.9353592Z ##[endgroup]
2022-12-01T10:21:11.9354320Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_x867oa81)
2022-12-01T10:21:11.9354677Z 
2022-12-01T10:41:27.5829121Z 
2022-12-01T10:41:27.5832132Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:41:27.5833100Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_go5u7ya9)
2022-12-01T10:41:27.5839145Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:41:27.5878041Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T10:41:27.5939081Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5942185Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5942983Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5943760Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5944542Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5945770Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5946675Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5947638Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5948518Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5950062Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5951096Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5952073Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5953006Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5953968Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5954805Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5955694Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5956566Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5957359Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5958159Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5959012Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5959895Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5960767Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5961532Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5962389Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5963192Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5963940Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5964685Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5965478Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5966237Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5966949Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5967645Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5968373Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5969090Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5969838Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5970593Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5971357Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5972116Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5972896Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5973650Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5974357Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5975098Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5975897Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5976771Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5977591Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5978563Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5979343Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5980154Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5981016Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5981810Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5982578Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5983354Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5984126Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5984900Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5985700Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5986498Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5987237Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5988016Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5988725Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5989951Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5990697Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5991460Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5992162Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5992870Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5993543Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5994217Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5994932Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5995743Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5996479Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5997124Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5997846Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5998545Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5999186Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.5999905Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6000689Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6001379Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6002029Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6002709Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6003443Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6004168Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6004894Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6005617Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6006272Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6006983Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6007755Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6008669Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6009408Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6010195Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6010783Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6011238Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6011665Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6012099Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6012538Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6012960Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6013407Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6013866Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6014323Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6014766Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6015216Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6015644Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6016020Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6016401Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6016765Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6017128Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6017489Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6017881Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6018270Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6018636Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6019042Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6019505Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6019896Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6020295Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6020700Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6021126Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6021521Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6021934Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6022351Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6022747Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6023176Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6023607Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6024023Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6024393Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6024763Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6025155Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6025529Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6025915Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6026309Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6026745Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6027260Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6027779Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6028194Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6028596Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6029028Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6030076Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6030623Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6031067Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6031491Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6031929Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6032319Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6032694Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6033087Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6033488Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6033900Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6034339Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6034758Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6035178Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6035587Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6036051Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6036535Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6037216Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6037851Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6038451Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6039038Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6039635Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6040237Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6040843Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6041426Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6041969Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6042458Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6042908Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6043278Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6043676Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6044083Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6044462Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6044980Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6045420Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6045864Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6046335Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6046752Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6047142Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6047541Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6047976Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6048404Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6048800Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6049220Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6049646Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6050076Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6050482Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6050891Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6051296Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6051698Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6052102Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6052517Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6052977Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6053442Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6053839Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6054237Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6054642Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6055059Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6055438Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6055787Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6056159Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6056536Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6056896Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6057265Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6057650Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6058035Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6058383Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6058739Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6059117Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6059496Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6059887Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6060259Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6060589Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6060968Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6061368Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6061781Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6062299Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6062755Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6063170Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6063576Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6064008Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6064432Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6064857Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6065258Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6065676Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6066090Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6066490Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6066889Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6067295Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6067747Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6068237Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6068703Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6069151Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6070142Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6070605Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6071035Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6071528Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6071975Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6072415Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6072866Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6073329Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6073842Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6074327Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6074730Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6075128Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6075535Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6075943Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6076313Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6076695Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6077084Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6077456Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6077829Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6078192Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6078551Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6078934Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6079337Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6079746Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6080190Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6080621Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6081008Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6081377Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6081797Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6082168Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6082572Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6082945Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6083326Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6083719Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6084083Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6084463Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6084834Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6085197Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6085621Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6086066Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6086477Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6086874Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6087279Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6087707Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6088099Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6088492Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6088906Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6089384Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6089809Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6090209Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6090601Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6091002Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6091383Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6091800Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6092230Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6092669Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6093407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6093854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6094435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6094904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6095135Z 
2022-12-01T10:41:27.6095246Z Running tests...
2022-12-01T10:41:27.6095639Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6128209Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6128814Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6129388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1485
2022-12-01T10:41:27.6130025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1487
2022-12-01T10:41:27.6130658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6131099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6131680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6132155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6132744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6133175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6133746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6134210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6134656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6135169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6135832Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6136529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6137037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6137513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6138043Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6139059Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6139725Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6140548Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6141167Z [1669890079.895887] [7ff5451e42d4:1485 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6141654Z [1669890079.895887] [7ff5451e42d4:1485 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6142115Z [1669890079.899562] [7ff5451e42d4:1487 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6142592Z [1669890079.899562] [7ff5451e42d4:1487 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6143123Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6143954Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6144594Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6145406Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6146060Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6146882Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6147589Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6148404Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6149058Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6150424Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6151066Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:41:27.6151883Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:41:27.6152366Z ok (7.342s)
2022-12-01T10:41:27.6152518Z 
2022-12-01T10:41:27.6152793Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6153108Z Ran 1 test in 7.342s
2022-12-01T10:41:27.6153271Z 
2022-12-01T10:41:27.6153365Z OK
2022-12-01T10:41:27.6153503Z 
2022-12-01T10:41:27.6153628Z Generating XML reports...
2022-12-01T10:41:27.6154219Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102113.xml
2022-12-01T10:41:27.6154940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6155396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6155974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6156528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6156775Z 
2022-12-01T10:41:27.6156887Z Running tests...
2022-12-01T10:41:27.6157299Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6157812Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6158349Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-12-01T10:41:27.6158660Z 
2022-12-01T10:41:27.6158929Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6159259Z Ran 1 test in 0.004s
2022-12-01T10:41:27.6159420Z 
2022-12-01T10:41:27.6159511Z OK (skipped=1)
2022-12-01T10:41:27.6159666Z 
2022-12-01T10:41:27.6159793Z Generating XML reports...
2022-12-01T10:41:27.6160402Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102123.xml
2022-12-01T10:41:27.6161133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6161569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6162148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6162620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6162850Z 
2022-12-01T10:41:27.6162940Z Running tests...
2022-12-01T10:41:27.6163345Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6163878Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6164385Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6164856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1925
2022-12-01T10:41:27.6165399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1926
2022-12-01T10:41:27.6166016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6166448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6167029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6167504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6168082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6168509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6169086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6169559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6170001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6170505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6171163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6171860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6172368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6172841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6173184Z ok (4.013s)
2022-12-01T10:41:27.6173335Z 
2022-12-01T10:41:27.6173606Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6173922Z Ran 1 test in 4.014s
2022-12-01T10:41:27.6174149Z 
2022-12-01T10:41:27.6174252Z OK
2022-12-01T10:41:27.6174391Z 
2022-12-01T10:41:27.6174517Z Generating XML reports...
2022-12-01T10:41:27.6175113Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102125.xml
2022-12-01T10:41:27.6175831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6176285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6176858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6177311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6177544Z 
2022-12-01T10:41:27.6177654Z Running tests...
2022-12-01T10:41:27.6178065Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6178592Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6179122Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6180174Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.622s)
2022-12-01T10:41:27.6180757Z 
2022-12-01T10:41:27.6181027Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6181358Z Ran 1 test in 1.622s
2022-12-01T10:41:27.6181503Z 
2022-12-01T10:41:27.6181614Z OK (skipped=1)
2022-12-01T10:41:27.6181769Z 
2022-12-01T10:41:27.6181898Z Generating XML reports...
2022-12-01T10:41:27.6182509Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml
2022-12-01T10:41:27.6183278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6183732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6184310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6184781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6185015Z 
2022-12-01T10:41:27.6185105Z Running tests...
2022-12-01T10:41:27.6185514Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6186046Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6186584Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6187091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2376
2022-12-01T10:41:27.6187543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2377
2022-12-01T10:41:27.6188150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6188582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6189157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6190161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6190759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6191187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6191761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6192325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6192779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6193284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6193945Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6194637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6195144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6195648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpapoy5i07
2022-12-01T10:41:27.6196196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpapoy5i07/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6196709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6197190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxivf9da
2022-12-01T10:41:27.6197723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxivf9da/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6198235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6198705Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6199190Z [1669890100.332812] [7ff5451e42d4:2377 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6199670Z [1669890100.332812] [7ff5451e42d4:2377 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6200229Z [1669890100.318908] [7ff5451e42d4:2376 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6200676Z [1669890100.318908] [7ff5451e42d4:2376 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6201019Z ok (6.063s)
2022-12-01T10:41:27.6201169Z 
2022-12-01T10:41:27.6201444Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6201778Z Ran 1 test in 6.063s
2022-12-01T10:41:27.6201922Z 
2022-12-01T10:41:27.6202015Z OK
2022-12-01T10:41:27.6202151Z 
2022-12-01T10:41:27.6202277Z Generating XML reports...
2022-12-01T10:41:27.6202884Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml
2022-12-01T10:41:27.6203585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6204043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6204626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6205099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6205311Z 
2022-12-01T10:41:27.6205424Z Running tests...
2022-12-01T10:41:27.6205830Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6206365Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6206901Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6207435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2789
2022-12-01T10:41:27.6207887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2790
2022-12-01T10:41:27.6208494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6208987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6209579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6210049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6210632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6211060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6211630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6212095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6212533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6213044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6213706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6214407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6214916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6215422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmf16tf8
2022-12-01T10:41:27.6215966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmf16tf8/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6216479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6216958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrwbpvia
2022-12-01T10:41:27.6217565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrwbpvia/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6218082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6218549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6219035Z [1669890108.715246] [7ff5451e42d4:2790 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6219513Z [1669890108.715246] [7ff5451e42d4:2790 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6219984Z [1669890108.697742] [7ff5451e42d4:2789 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6220433Z [1669890108.697742] [7ff5451e42d4:2789 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6220783Z ok (6.107s)
2022-12-01T10:41:27.6220933Z 
2022-12-01T10:41:27.6221212Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6221546Z Ran 1 test in 6.107s
2022-12-01T10:41:27.6221689Z 
2022-12-01T10:41:27.6221783Z OK
2022-12-01T10:41:27.6221917Z 
2022-12-01T10:41:27.6222043Z Generating XML reports...
2022-12-01T10:41:27.6222645Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102143.xml
2022-12-01T10:41:27.6223341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6223794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6224372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6224845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6225059Z 
2022-12-01T10:41:27.6225170Z Running tests...
2022-12-01T10:41:27.6225627Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6226166Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6226700Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6227228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3130
2022-12-01T10:41:27.6227683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3131
2022-12-01T10:41:27.6228293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6228725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6229302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6230476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6231055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6231503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6232075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6232536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6232972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6233472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6234128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6234947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6235453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6235926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6236427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq3bp4tiv
2022-12-01T10:41:27.6236954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq3bp4tiv/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6237488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasdf22qr
2022-12-01T10:41:27.6238026Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasdf22qr/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6238538Z [1669890116.974999] [7ff5451e42d4:3130 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6239009Z [1669890116.974999] [7ff5451e42d4:3130 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6239478Z [1669890116.981506] [7ff5451e42d4:3131 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6239953Z [1669890116.981506] [7ff5451e42d4:3131 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6240428Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6240898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6241378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6241860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6242345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6242911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6243402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6243874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6244332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6244804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6245280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6245753Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6246083Z ok (7.218s)
2022-12-01T10:41:27.6246233Z 
2022-12-01T10:41:27.6246522Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6246857Z Ran 1 test in 7.218s
2022-12-01T10:41:27.6247021Z 
2022-12-01T10:41:27.6247098Z OK
2022-12-01T10:41:27.6247234Z 
2022-12-01T10:41:27.6247362Z Generating XML reports...
2022-12-01T10:41:27.6247972Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102151.xml
2022-12-01T10:41:27.6248695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6249130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6249706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6250173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6250404Z 
2022-12-01T10:41:27.6250496Z Running tests...
2022-12-01T10:41:27.6250905Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6251505Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6252070Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6252590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3487
2022-12-01T10:41:27.6253038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3488
2022-12-01T10:41:27.6253645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6254078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6254654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6255124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6255710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6256141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6256715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6257183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6257619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6258121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6258778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6259473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6260049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6260533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6261033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzndrhj2
2022-12-01T10:41:27.6261577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzndrhj2/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6262093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9zzcatz
2022-12-01T10:41:27.6262631Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9zzcatz/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6263296Z [1669890126.488466] [7ff5451e42d4:3487 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6263912Z [1669890126.488466] [7ff5451e42d4:3487 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6264383Z [1669890126.489598] [7ff5451e42d4:3488 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6264951Z [1669890126.489598] [7ff5451e42d4:3488 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6265491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6266054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6266520Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6266999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6267347Z ok (6.214s)
2022-12-01T10:41:27.6267499Z 
2022-12-01T10:41:27.6267770Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6268194Z Ran 1 test in 6.214s
2022-12-01T10:41:27.6268358Z 
2022-12-01T10:41:27.6268457Z OK
2022-12-01T10:41:27.6268595Z 
2022-12-01T10:41:27.6268701Z Generating XML reports...
2022-12-01T10:41:27.6269318Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102201.xml
2022-12-01T10:41:27.6270298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6270749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6271314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6271788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6272019Z 
2022-12-01T10:41:27.6272129Z Running tests...
2022-12-01T10:41:27.6272537Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6273061Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6273634Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6274179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3831
2022-12-01T10:41:27.6274610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3832
2022-12-01T10:41:27.6275214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6275669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6276243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6276696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6277365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6277822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6278380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6278844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6279298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6279799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6280492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6281188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6281725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6282199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6282682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpor_mxwti
2022-12-01T10:41:27.6283227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpor_mxwti/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6283760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw7r6nww
2022-12-01T10:41:27.6284276Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw7r6nww/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6284785Z [1669890134.834495] [7ff5451e42d4:3832 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6285267Z [1669890134.834495] [7ff5451e42d4:3832 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6285850Z [1669890134.826007] [7ff5451e42d4:3831 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6286304Z [1669890134.826007] [7ff5451e42d4:3831 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6286781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6287268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6287749Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6288206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6288555Z ok (6.158s)
2022-12-01T10:41:27.6288707Z 
2022-12-01T10:41:27.6288988Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6289307Z Ran 1 test in 6.158s
2022-12-01T10:41:27.6289472Z 
2022-12-01T10:41:27.6289566Z OK
2022-12-01T10:41:27.6289703Z 
2022-12-01T10:41:27.6289828Z Generating XML reports...
2022-12-01T10:41:27.6290440Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102209.xml
2022-12-01T10:41:27.6291138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6291590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6292171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6292626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6292859Z 
2022-12-01T10:41:27.6292969Z Running tests...
2022-12-01T10:41:27.6293381Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6293965Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6294588Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6295160Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4121
2022-12-01T10:41:27.6295610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4122
2022-12-01T10:41:27.6296220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6296654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6297232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6297706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6298270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6298725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6299295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6299759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6300194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6300697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6301356Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6302046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6302612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6303093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6303597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirfjij8y
2022-12-01T10:41:27.6304122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirfjij8y/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6304655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_fd3jiv
2022-12-01T10:41:27.6305190Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_fd3jiv/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6305703Z [1669890143.349009] [7ff5451e42d4:4122 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6306167Z [1669890143.349009] [7ff5451e42d4:4122 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6306647Z [1669890143.340892] [7ff5451e42d4:4121 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6307123Z [1669890143.340892] [7ff5451e42d4:4121 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6307602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6308072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6308429Z ok (6.395s)
2022-12-01T10:41:27.6308578Z 
2022-12-01T10:41:27.6308858Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6309171Z Ran 1 test in 6.395s
2022-12-01T10:41:27.6309333Z 
2022-12-01T10:41:27.6309671Z OK
2022-12-01T10:41:27.6309816Z 
2022-12-01T10:41:27.6309946Z Generating XML reports...
2022-12-01T10:41:27.6310561Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102218.xml
2022-12-01T10:41:27.6311345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6311815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6312400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6312852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6313084Z 
2022-12-01T10:41:27.6313194Z Running tests...
2022-12-01T10:41:27.6313610Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6314144Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6314710Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6315278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4470
2022-12-01T10:41:27.6315730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4471
2022-12-01T10:41:27.6316340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6316779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6317362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6317835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6318394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6318844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6319416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6319968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6320406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6320905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6321566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6322240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6322764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6323240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6323745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_54km4t
2022-12-01T10:41:27.6324274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_54km4t/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6324806Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy1wvujs
2022-12-01T10:41:27.6325342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy1wvujs/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6325854Z [1669890151.914679] [7ff5451e42d4:4471 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6326313Z [1669890151.914679] [7ff5451e42d4:4471 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6326791Z [1669890151.914722] [7ff5451e42d4:4470 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6327263Z [1669890151.914722] [7ff5451e42d4:4470 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6327889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6328365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6328850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6329333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6329666Z ok (7.138s)
2022-12-01T10:41:27.6329815Z 
2022-12-01T10:41:27.6330091Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6330422Z Ran 1 test in 7.139s
2022-12-01T10:41:27.6330586Z 
2022-12-01T10:41:27.6330683Z OK
2022-12-01T10:41:27.6330797Z 
2022-12-01T10:41:27.6330922Z Generating XML reports...
2022-12-01T10:41:27.6331532Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102226.xml
2022-12-01T10:41:27.6332256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6332689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6333272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6333748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6333978Z 
2022-12-01T10:41:27.6334090Z Running tests...
2022-12-01T10:41:27.6334482Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6335017Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6335585Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6336173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4853
2022-12-01T10:41:27.6336630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4854
2022-12-01T10:41:27.6337237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6337695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6338257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6338732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6339319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6339767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6340323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6340796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6341250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6341732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6342391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6343087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6343611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6344068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6344563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5yd3ib9_
2022-12-01T10:41:27.6345181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5yd3ib9_/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6345723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_rg98xd
2022-12-01T10:41:27.6346240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_rg98xd/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6346749Z [1669890161.355174] [7ff5451e42d4:4853 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6347228Z [1669890161.355174] [7ff5451e42d4:4853 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6347684Z [1669890161.362261] [7ff5451e42d4:4854 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6348154Z [1669890161.362261] [7ff5451e42d4:4854 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6348640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6349129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6349823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6350306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6350655Z ok (6.656s)
2022-12-01T10:41:27.6350806Z 
2022-12-01T10:41:27.6351088Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6351405Z Ran 1 test in 6.656s
2022-12-01T10:41:27.6351567Z 
2022-12-01T10:41:27.6351663Z OK
2022-12-01T10:41:27.6351798Z 
2022-12-01T10:41:27.6351923Z Generating XML reports...
2022-12-01T10:41:27.6352521Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102236.xml
2022-12-01T10:41:27.6353345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6353802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6354382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6354839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6355071Z 
2022-12-01T10:41:27.6355181Z Running tests...
2022-12-01T10:41:27.6355592Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6356108Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6356696Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6357256Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5206
2022-12-01T10:41:27.6357720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5207
2022-12-01T10:41:27.6358315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6358773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6359352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6359806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6360385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6360830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6361404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6361921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6362386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6362889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6363553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6364225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6364749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6365222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6365710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3oypvhu
2022-12-01T10:41:27.6366267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3oypvhu/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6366803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0z_1hk87
2022-12-01T10:41:27.6367341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0z_1hk87/_remote_module_non_scriptable.py
2022-12-01T10:41:27.6367831Z [1669890170.095440] [7ff5451e42d4:5206 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6368315Z [1669890170.095440] [7ff5451e42d4:5206 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6368793Z [1669890170.104491] [7ff5451e42d4:5207 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6369262Z [1669890170.104491] [7ff5451e42d4:5207 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6369799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6370287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6370766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6371253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.6371587Z ok (5.979s)
2022-12-01T10:41:27.6371736Z 
2022-12-01T10:41:27.6372015Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6372349Z Ran 1 test in 5.979s
2022-12-01T10:41:27.6372513Z 
2022-12-01T10:41:27.6372589Z OK
2022-12-01T10:41:27.6372724Z 
2022-12-01T10:41:27.6372848Z Generating XML reports...
2022-12-01T10:41:27.6373524Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102245.xml
2022-12-01T10:41:27.6374256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6374695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6375278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6375753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6375984Z 
2022-12-01T10:41:27.6376074Z Running tests...
2022-12-01T10:41:27.6376478Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6377006Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6377562Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6378698Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.613s)
2022-12-01T10:41:27.6379241Z 
2022-12-01T10:41:27.6379512Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6379840Z Ran 1 test in 1.614s
2022-12-01T10:41:27.6380002Z 
2022-12-01T10:41:27.6380111Z OK (skipped=1)
2022-12-01T10:41:27.6380249Z 
2022-12-01T10:41:27.6380376Z Generating XML reports...
2022-12-01T10:41:27.6381032Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102253.xml
2022-12-01T10:41:27.6381745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6382194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6382754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6383237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6383468Z 
2022-12-01T10:41:27.6383579Z Running tests...
2022-12-01T10:41:27.6383970Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6384498Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6385048Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6385575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5642
2022-12-01T10:41:27.6386007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5643
2022-12-01T10:41:27.6386612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6387139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6387702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6388173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6388757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6389203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6389962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6390430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6390888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6391374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6392044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6392737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6393262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6393719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6394058Z ok (3.964s)
2022-12-01T10:41:27.6394208Z 
2022-12-01T10:41:27.6394480Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6394815Z Ran 1 test in 3.964s
2022-12-01T10:41:27.6394959Z 
2022-12-01T10:41:27.6395053Z OK
2022-12-01T10:41:27.6395190Z 
2022-12-01T10:41:27.6395316Z Generating XML reports...
2022-12-01T10:41:27.6395922Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102257.xml
2022-12-01T10:41:27.6396705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6397171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6397748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6398221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6398434Z 
2022-12-01T10:41:27.6398544Z Running tests...
2022-12-01T10:41:27.6398955Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6399489Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6400035Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6401115Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.723s)
2022-12-01T10:41:27.6401632Z 
2022-12-01T10:41:27.6401898Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6402228Z Ran 1 test in 1.723s
2022-12-01T10:41:27.6402391Z 
2022-12-01T10:41:27.6402480Z OK (skipped=1)
2022-12-01T10:41:27.6402636Z 
2022-12-01T10:41:27.6402763Z Generating XML reports...
2022-12-01T10:41:27.6403369Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102303.xml
2022-12-01T10:41:27.6404083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6404597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6405188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6405658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6405891Z 
2022-12-01T10:41:27.6406002Z Running tests...
2022-12-01T10:41:27.6406392Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6406918Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6407448Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6407935Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6070
2022-12-01T10:41:27.6408386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6071
2022-12-01T10:41:27.6408998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6409455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6410015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6410482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6411067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6411495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6412067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6412534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6412988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6413523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6414188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6414878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6415402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6415859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6416332Z [1669890192.483757] [7ff5451e42d4:6070 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6416811Z [1669890192.483757] [7ff5451e42d4:6070 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6417272Z [1669890192.490456] [7ff5451e42d4:6071 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6417743Z [1669890192.490456] [7ff5451e42d4:6071 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6418089Z ok (5.957s)
2022-12-01T10:41:27.6418238Z 
2022-12-01T10:41:27.6418512Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6418825Z Ran 1 test in 5.957s
2022-12-01T10:41:27.6418988Z 
2022-12-01T10:41:27.6419080Z OK
2022-12-01T10:41:27.6419213Z 
2022-12-01T10:41:27.6419338Z Generating XML reports...
2022-12-01T10:41:27.6419928Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102307.xml
2022-12-01T10:41:27.6420646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6421156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6421740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6422197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6422428Z 
2022-12-01T10:41:27.6422539Z Running tests...
2022-12-01T10:41:27.6422949Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6423484Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6423972Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T10:41:27.6424260Z 
2022-12-01T10:41:27.6424614Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6424977Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6425140Z 
2022-12-01T10:41:27.6425231Z OK (skipped=1)
2022-12-01T10:41:27.6425489Z 
2022-12-01T10:41:27.6425669Z Generating XML reports...
2022-12-01T10:41:27.6426285Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102315.xml
2022-12-01T10:41:27.6427004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6427436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6428018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6428486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6428716Z 
2022-12-01T10:41:27.6428806Z Running tests...
2022-12-01T10:41:27.6429212Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6429966Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6430435Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6431008Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T10:41:27.6431326Z 
2022-12-01T10:41:27.6431604Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6431928Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6432090Z 
2022-12-01T10:41:27.6432180Z OK (skipped=1)
2022-12-01T10:41:27.6432338Z 
2022-12-01T10:41:27.6432465Z Generating XML reports...
2022-12-01T10:41:27.6433345Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102317.xml
2022-12-01T10:41:27.6434091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6434523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6435103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6435583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6435818Z 
2022-12-01T10:41:27.6435930Z Running tests...
2022-12-01T10:41:27.6436320Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6436853Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6437331Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6437838Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T10:41:27.6438146Z 
2022-12-01T10:41:27.6438416Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6438808Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6438973Z 
2022-12-01T10:41:27.6439085Z OK (skipped=1)
2022-12-01T10:41:27.6439333Z 
2022-12-01T10:41:27.6439459Z Generating XML reports...
2022-12-01T10:41:27.6440076Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102319.xml
2022-12-01T10:41:27.6440794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6441229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6441814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6442279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6442510Z 
2022-12-01T10:41:27.6442620Z Running tests...
2022-12-01T10:41:27.6443005Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6443537Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6444044Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6444589Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T10:41:27.6444919Z 
2022-12-01T10:41:27.6445185Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6445512Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6445674Z 
2022-12-01T10:41:27.6445783Z OK (skipped=1)
2022-12-01T10:41:27.6445937Z 
2022-12-01T10:41:27.6446044Z Generating XML reports...
2022-12-01T10:41:27.6446649Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102321.xml
2022-12-01T10:41:27.6447364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6447820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6448447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6448926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6449157Z 
2022-12-01T10:41:27.6449267Z Running tests...
2022-12-01T10:41:27.6449658Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6450189Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6450665Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:41:27.6451184Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T10:41:27.6451487Z 
2022-12-01T10:41:27.6451738Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6452065Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6452231Z 
2022-12-01T10:41:27.6452339Z OK (skipped=1)
2022-12-01T10:41:27.6452493Z 
2022-12-01T10:41:27.6452602Z Generating XML reports...
2022-12-01T10:41:27.6453210Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102324.xml
2022-12-01T10:41:27.6453922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6454376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6454933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6455405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6455639Z 
2022-12-01T10:41:27.6455749Z Running tests...
2022-12-01T10:41:27.6456156Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6456670Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6457229Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6457709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6923
2022-12-01T10:41:27.6458143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6924
2022-12-01T10:41:27.6458750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6459205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6459789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6460245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6460830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6461283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6461843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6462313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6462769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6463270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6463911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6464602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6465123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6465762Z STAGE:2022-12-01 10:23:29 6924:6924 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6466228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6466808Z STAGE:2022-12-01 10:23:29 6923:6923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6467292Z [1669890211.663527] [7ff5451e42d4:6924 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6467757Z [1669890211.663527] [7ff5451e42d4:6924 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6468232Z [1669890211.677478] [7ff5451e42d4:6923 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6468701Z [1669890211.677478] [7ff5451e42d4:6923 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6469760Z STAGE:2022-12-01 10:23:32 6924:6924 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:23:32 6923:6923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6470155Z 
2022-12-01T10:41:27.6470495Z STAGE:2022-12-01 10:23:32 6924:6924 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6471049Z STAGE:2022-12-01 10:23:32 6923:6923 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6471621Z STAGE:2022-12-01 10:23:32 6924:6924 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6472200Z STAGE:2022-12-01 10:23:32 6923:6923 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6472535Z ok (7.099s)
2022-12-01T10:41:27.6472684Z 
2022-12-01T10:41:27.6472959Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6473291Z Ran 1 test in 7.099s
2022-12-01T10:41:27.6473455Z 
2022-12-01T10:41:27.6473637Z OK
2022-12-01T10:41:27.6473759Z 
2022-12-01T10:41:27.6473885Z Generating XML reports...
2022-12-01T10:41:27.6474493Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102326.xml
2022-12-01T10:41:27.6475211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6475646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6476227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6476695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6476925Z 
2022-12-01T10:41:27.6477036Z Running tests...
2022-12-01T10:41:27.6477426Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6477957Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6478506Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T10:41:27.6478827Z 
2022-12-01T10:41:27.6479097Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6479408Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6479572Z 
2022-12-01T10:41:27.6479680Z OK (skipped=1)
2022-12-01T10:41:27.6479835Z 
2022-12-01T10:41:27.6479959Z Generating XML reports...
2022-12-01T10:41:27.6480591Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102335.xml
2022-12-01T10:41:27.6481311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6481763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6482347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6482877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6483121Z 
2022-12-01T10:41:27.6483232Z Running tests...
2022-12-01T10:41:27.6483645Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6484159Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6484702Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T10:41:27.6485023Z 
2022-12-01T10:41:27.6485293Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6485624Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6485766Z 
2022-12-01T10:41:27.6485877Z OK (skipped=1)
2022-12-01T10:41:27.6486032Z 
2022-12-01T10:41:27.6486157Z Generating XML reports...
2022-12-01T10:41:27.6486756Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102337.xml
2022-12-01T10:41:27.6487466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6487920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6488501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6488970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6489200Z 
2022-12-01T10:41:27.6489291Z Running tests...
2022-12-01T10:41:27.6489695Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6490226Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6490745Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T10:41:27.6491123Z 
2022-12-01T10:41:27.6491397Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6491725Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6491886Z 
2022-12-01T10:41:27.6491993Z OK (skipped=1)
2022-12-01T10:41:27.6492150Z 
2022-12-01T10:41:27.6492261Z Generating XML reports...
2022-12-01T10:41:27.6492868Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102339.xml
2022-12-01T10:41:27.6493582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6494035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6494594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6495069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6495298Z 
2022-12-01T10:41:27.6495415Z Running tests...
2022-12-01T10:41:27.6495807Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6496338Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6496873Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T10:41:27.6497191Z 
2022-12-01T10:41:27.6497456Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6497769Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6497935Z 
2022-12-01T10:41:27.6498048Z OK (skipped=1)
2022-12-01T10:41:27.6498203Z 
2022-12-01T10:41:27.6498328Z Generating XML reports...
2022-12-01T10:41:27.6498911Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102342.xml
2022-12-01T10:41:27.6499631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6499872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6500269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6500466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6500486Z 
2022-12-01T10:41:27.6500596Z Running tests...
2022-12-01T10:41:27.6500859Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6501152Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6501450Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.003s)
2022-12-01T10:41:27.6501470Z 
2022-12-01T10:41:27.6501747Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6501858Z Ran 1 test in 0.003s
2022-12-01T10:41:27.6501882Z 
2022-12-01T10:41:27.6501990Z OK (skipped=1)
2022-12-01T10:41:27.6502009Z 
2022-12-01T10:41:27.6502136Z Generating XML reports...
2022-12-01T10:41:27.6502589Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102344.xml
2022-12-01T10:41:27.6502962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6503138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6503500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6503695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6503717Z 
2022-12-01T10:41:27.6503828Z Running tests...
2022-12-01T10:41:27.6504094Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6504409Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6504778Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6504997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7816
2022-12-01T10:41:27.6505214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7817
2022-12-01T10:41:27.6505588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6505748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6506131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6506324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6506692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6506874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6507254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6507446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6507698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6507925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6508331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6508729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6508963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6509534Z STAGE:2022-12-01 10:23:49 7817:7817 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6509789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6510136Z STAGE:2022-12-01 10:23:49 7816:7816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6510376Z [1669890231.459654] [7ff5451e42d4:7817 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6510616Z [1669890231.459654] [7ff5451e42d4:7817 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6510829Z [1669890231.486313] [7ff5451e42d4:7816 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6511072Z [1669890231.486313] [7ff5451e42d4:7816 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6511628Z STAGE:2022-12-01 10:23:51 7817:7817 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:23:51 7816:7816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6511650Z 
2022-12-01T10:41:27.6511980Z STAGE:2022-12-01 10:23:51 7817:7817 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6512306Z STAGE:2022-12-01 10:23:51 7816:7816 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6512639Z STAGE:2022-12-01 10:23:51 7817:7817 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6512969Z STAGE:2022-12-01 10:23:51 7816:7816 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6513075Z ok (7.004s)
2022-12-01T10:41:27.6513095Z 
2022-12-01T10:41:27.6513363Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6513478Z Ran 1 test in 7.005s
2022-12-01T10:41:27.6513573Z 
2022-12-01T10:41:27.6513655Z OK
2022-12-01T10:41:27.6513674Z 
2022-12-01T10:41:27.6513805Z Generating XML reports...
2022-12-01T10:41:27.6514260Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102346.xml
2022-12-01T10:41:27.6514632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6514811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6515194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6515390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6515410Z 
2022-12-01T10:41:27.6515519Z Running tests...
2022-12-01T10:41:27.6515792Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6516085Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6516357Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:41:27.6516377Z 
2022-12-01T10:41:27.6516642Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6516754Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6516774Z 
2022-12-01T10:41:27.6516883Z OK (skipped=1)
2022-12-01T10:41:27.6516903Z 
2022-12-01T10:41:27.6517027Z Generating XML reports...
2022-12-01T10:41:27.6517475Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102355.xml
2022-12-01T10:41:27.6517851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6518011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6518391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6518645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6518667Z 
2022-12-01T10:41:27.6518782Z Running tests...
2022-12-01T10:41:27.6519046Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6519359Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6519637Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:41:27.6519659Z 
2022-12-01T10:41:27.6519928Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6520042Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6520062Z 
2022-12-01T10:41:27.6520154Z OK (skipped=1)
2022-12-01T10:41:27.6520173Z 
2022-12-01T10:41:27.6520300Z Generating XML reports...
2022-12-01T10:41:27.6520747Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102357.xml
2022-12-01T10:41:27.6521127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6521303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6521684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6521879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6521899Z 
2022-12-01T10:41:27.6522010Z Running tests...
2022-12-01T10:41:27.6522274Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6522564Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6522837Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6523060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8427
2022-12-01T10:41:27.6523337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8428
2022-12-01T10:41:27.6523708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6523886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6524266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6524458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6524803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6524978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6525349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6525546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6525792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6526035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6526440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6526840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6527074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6527295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.6527526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6527819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.6528230Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6528626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6528959Z STAGE:2022-12-01 10:24:03 8427:8427 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6529287Z STAGE:2022-12-01 10:24:03 8428:8428 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6529527Z [1669890245.031980] [7ff5451e42d4:8428 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6529773Z [1669890245.031980] [7ff5451e42d4:8428 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6529987Z [1669890245.021127] [7ff5451e42d4:8427 :0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6530226Z [1669890245.021127] [7ff5451e42d4:8427 :0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6530774Z STAGE:2022-12-01 10:24:05 8428:8428 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:05 8427:8427 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6530796Z 
2022-12-01T10:41:27.6531126Z STAGE:2022-12-01 10:24:05 8428:8428 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6531455Z STAGE:2022-12-01 10:24:05 8427:8427 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6531789Z STAGE:2022-12-01 10:24:05 8428:8428 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6532116Z STAGE:2022-12-01 10:24:05 8427:8427 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6532280Z ok (7.263s)
2022-12-01T10:41:27.6532300Z 
2022-12-01T10:41:27.6532569Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6532683Z Ran 1 test in 7.263s
2022-12-01T10:41:27.6532703Z 
2022-12-01T10:41:27.6532777Z OK
2022-12-01T10:41:27.6532796Z 
2022-12-01T10:41:27.6532921Z Generating XML reports...
2022-12-01T10:41:27.6533369Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102359.xml
2022-12-01T10:41:27.6533746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6533926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6534307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6534501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6534524Z 
2022-12-01T10:41:27.6534638Z Running tests...
2022-12-01T10:41:27.6534885Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6535202Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6535460Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6535679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8882
2022-12-01T10:41:27.6535903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8883
2022-12-01T10:41:27.6536274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6536457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6536839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6537082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6537443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6537618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6537993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6538182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6538430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6538674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6539075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6539482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6539718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6539928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6540087Z skip: Skipped due to small world size. (4.035s)
2022-12-01T10:41:27.6540107Z 
2022-12-01T10:41:27.6540373Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6540486Z Ran 1 test in 4.035s
2022-12-01T10:41:27.6540507Z 
2022-12-01T10:41:27.6540616Z OK (skipped=1)
2022-12-01T10:41:27.6540635Z 
2022-12-01T10:41:27.6540758Z Generating XML reports...
2022-12-01T10:41:27.6541206Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102409.xml
2022-12-01T10:41:27.6541648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6541805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6542188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6542380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6542399Z 
2022-12-01T10:41:27.6542507Z Running tests...
2022-12-01T10:41:27.6542772Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6543083Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6543381Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:41:27.6543402Z 
2022-12-01T10:41:27.6543666Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6543786Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6543809Z 
2022-12-01T10:41:27.6543898Z OK (skipped=1)
2022-12-01T10:41:27.6543917Z 
2022-12-01T10:41:27.6544043Z Generating XML reports...
2022-12-01T10:41:27.6544492Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102415.xml
2022-12-01T10:41:27.6544866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6545044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6545427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6545621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6545641Z 
2022-12-01T10:41:27.6545751Z Running tests...
2022-12-01T10:41:27.6546017Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6546364Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6546676Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:41:27.6546696Z 
2022-12-01T10:41:27.6546961Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6547072Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6547092Z 
2022-12-01T10:41:27.6547199Z OK (skipped=1)
2022-12-01T10:41:27.6547219Z 
2022-12-01T10:41:27.6547345Z Generating XML reports...
2022-12-01T10:41:27.6547790Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102417.xml
2022-12-01T10:41:27.6548195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6548373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6548744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6548937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6548957Z 
2022-12-01T10:41:27.6549066Z Running tests...
2022-12-01T10:41:27.6549330Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6549841Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6550129Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:41:27.6550149Z 
2022-12-01T10:41:27.6550414Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6550526Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6550546Z 
2022-12-01T10:41:27.6550634Z OK (skipped=1)
2022-12-01T10:41:27.6550747Z 
2022-12-01T10:41:27.6550861Z Generating XML reports...
2022-12-01T10:41:27.6551314Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102419.xml
2022-12-01T10:41:27.6551685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6551862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6552240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6552432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6552452Z 
2022-12-01T10:41:27.6552561Z Running tests...
2022-12-01T10:41:27.6552831Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6553125Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6553434Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:41:27.6553454Z 
2022-12-01T10:41:27.6553719Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6553833Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6553852Z 
2022-12-01T10:41:27.6553960Z OK (skipped=1)
2022-12-01T10:41:27.6553979Z 
2022-12-01T10:41:27.6554103Z Generating XML reports...
2022-12-01T10:41:27.6554550Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102421.xml
2022-12-01T10:41:27.6554921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6555099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6555460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6555658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6555735Z 
2022-12-01T10:41:27.6555852Z Running tests...
2022-12-01T10:41:27.6556119Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6556430Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6556852Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.6556872Z 
2022-12-01T10:41:27.6557136Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6557248Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6557267Z 
2022-12-01T10:41:27.6557375Z OK (skipped=1)
2022-12-01T10:41:27.6557394Z 
2022-12-01T10:41:27.6557499Z Generating XML reports...
2022-12-01T10:41:27.6557949Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102423.xml
2022-12-01T10:41:27.6558331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6558507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6558885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6559078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6559098Z 
2022-12-01T10:41:27.6559207Z Running tests...
2022-12-01T10:41:27.6559471Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6559766Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6560184Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.6560251Z 
2022-12-01T10:41:27.6560521Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6560637Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6560657Z 
2022-12-01T10:41:27.6560764Z OK (skipped=1)
2022-12-01T10:41:27.6560783Z 
2022-12-01T10:41:27.6560905Z Generating XML reports...
2022-12-01T10:41:27.6561351Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102425.xml
2022-12-01T10:41:27.6561724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6561901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6562266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6562460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6562480Z 
2022-12-01T10:41:27.6562590Z Running tests...
2022-12-01T10:41:27.6562859Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6563175Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6563438Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-12-01T10:41:27.6563458Z 
2022-12-01T10:41:27.6563720Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6563835Z Ran 1 test in 0.003s
2022-12-01T10:41:27.6563854Z 
2022-12-01T10:41:27.6563964Z OK (skipped=1)
2022-12-01T10:41:27.6563984Z 
2022-12-01T10:41:27.6564090Z Generating XML reports...
2022-12-01T10:41:27.6564537Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102427.xml
2022-12-01T10:41:27.6564911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6565096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6565544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6565744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6565765Z 
2022-12-01T10:41:27.6565875Z Running tests...
2022-12-01T10:41:27.6566143Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6566456Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6566864Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6566884Z 
2022-12-01T10:41:27.6567147Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6567263Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6567282Z 
2022-12-01T10:41:27.6567395Z OK (skipped=1)
2022-12-01T10:41:27.6567415Z 
2022-12-01T10:41:27.6567537Z Generating XML reports...
2022-12-01T10:41:27.6567984Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102429.xml
2022-12-01T10:41:27.6568356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6568532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6568914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6569090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6569110Z 
2022-12-01T10:41:27.6569219Z Running tests...
2022-12-01T10:41:27.6569482Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6569794Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6570278Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6570299Z 
2022-12-01T10:41:27.6570556Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6570669Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6570688Z 
2022-12-01T10:41:27.6570796Z OK (skipped=1)
2022-12-01T10:41:27.6570815Z 
2022-12-01T10:41:27.6570921Z Generating XML reports...
2022-12-01T10:41:27.6571370Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102431.xml
2022-12-01T10:41:27.6571738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6571915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6572294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6572495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6572515Z 
2022-12-01T10:41:27.6572625Z Running tests...
2022-12-01T10:41:27.6572890Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6573201Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6573609Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6573649Z 
2022-12-01T10:41:27.6573897Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6574011Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6574030Z 
2022-12-01T10:41:27.6574138Z OK (skipped=1)
2022-12-01T10:41:27.6574157Z 
2022-12-01T10:41:27.6574282Z Generating XML reports...
2022-12-01T10:41:27.6574729Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102434.xml
2022-12-01T10:41:27.6575148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6575331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6575713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6575886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6575928Z 
2022-12-01T10:41:27.6576020Z Running tests...
2022-12-01T10:41:27.6576286Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6576599Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6577023Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6577046Z 
2022-12-01T10:41:27.6577318Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6577432Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6577451Z 
2022-12-01T10:41:27.6577561Z OK (skipped=1)
2022-12-01T10:41:27.6577581Z 
2022-12-01T10:41:27.6577706Z Generating XML reports...
2022-12-01T10:41:27.6578133Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102436.xml
2022-12-01T10:41:27.6578503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6578681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6579067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6579260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6579324Z 
2022-12-01T10:41:27.6579439Z Running tests...
2022-12-01T10:41:27.6579708Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6580021Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6580416Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6580456Z 
2022-12-01T10:41:27.6580748Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6580863Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6580882Z 
2022-12-01T10:41:27.6580989Z OK (skipped=1)
2022-12-01T10:41:27.6581008Z 
2022-12-01T10:41:27.6581132Z Generating XML reports...
2022-12-01T10:41:27.6581587Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102438.xml
2022-12-01T10:41:27.6581958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6582139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6582523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6582697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6582739Z 
2022-12-01T10:41:27.6582829Z Running tests...
2022-12-01T10:41:27.6583097Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6583411Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6583826Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6583846Z 
2022-12-01T10:41:27.6584109Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6584224Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6584244Z 
2022-12-01T10:41:27.6584352Z OK (skipped=1)
2022-12-01T10:41:27.6584420Z 
2022-12-01T10:41:27.6584552Z Generating XML reports...
2022-12-01T10:41:27.6584979Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102440.xml
2022-12-01T10:41:27.6585353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6585531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6585915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6586111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6586131Z 
2022-12-01T10:41:27.6586242Z Running tests...
2022-12-01T10:41:27.6586509Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6586829Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6587256Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6587277Z 
2022-12-01T10:41:27.6587518Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6587633Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6587653Z 
2022-12-01T10:41:27.6587762Z OK (skipped=1)
2022-12-01T10:41:27.6587781Z 
2022-12-01T10:41:27.6587904Z Generating XML reports...
2022-12-01T10:41:27.6588349Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102442.xml
2022-12-01T10:41:27.6588725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6588903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6589542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6589849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6589870Z 
2022-12-01T10:41:27.6589961Z Running tests...
2022-12-01T10:41:27.6590232Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6590547Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6590960Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6590980Z 
2022-12-01T10:41:27.6591246Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6591361Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6591381Z 
2022-12-01T10:41:27.6591487Z OK (skipped=1)
2022-12-01T10:41:27.6591507Z 
2022-12-01T10:41:27.6591630Z Generating XML reports...
2022-12-01T10:41:27.6592065Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102444.xml
2022-12-01T10:41:27.6592437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6592614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6592996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6593189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6593209Z 
2022-12-01T10:41:27.6593317Z Running tests...
2022-12-01T10:41:27.6593581Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6593892Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6594291Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6594315Z 
2022-12-01T10:41:27.6594653Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6594757Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6594777Z 
2022-12-01T10:41:27.6594885Z OK (skipped=1)
2022-12-01T10:41:27.6594904Z 
2022-12-01T10:41:27.6595026Z Generating XML reports...
2022-12-01T10:41:27.6595477Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102446.xml
2022-12-01T10:41:27.6595849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6596026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6596404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6596595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6596618Z 
2022-12-01T10:41:27.6596707Z Running tests...
2022-12-01T10:41:27.6596975Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6597289Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6597591Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6597815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10659
2022-12-01T10:41:27.6598032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10660
2022-12-01T10:41:27.6598408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6598586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6598969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6599215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6599590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6599765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6600140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6600328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6600577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6600824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6601227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6601615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6601846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6602622Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:41:27.6602740Z   warnings.warn(
2022-12-01T10:41:27.6602970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6603706Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:41:27.6603870Z   warnings.warn(
2022-12-01T10:41:27.6603977Z ok (4.038s)
2022-12-01T10:41:27.6603998Z 
2022-12-01T10:41:27.6604267Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6604382Z Ran 1 test in 4.038s
2022-12-01T10:41:27.6604401Z 
2022-12-01T10:41:27.6604476Z OK
2022-12-01T10:41:27.6604495Z 
2022-12-01T10:41:27.6604620Z Generating XML reports...
2022-12-01T10:41:27.6605066Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102448.xml
2022-12-01T10:41:27.6605439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6605614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6606000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6606202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6606222Z 
2022-12-01T10:41:27.6606332Z Running tests...
2022-12-01T10:41:27.6606582Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6606902Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6607301Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6607321Z 
2022-12-01T10:41:27.6607585Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6607698Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6607717Z 
2022-12-01T10:41:27.6607828Z OK (skipped=1)
2022-12-01T10:41:27.6607847Z 
2022-12-01T10:41:27.6607971Z Generating XML reports...
2022-12-01T10:41:27.6608416Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102455.xml
2022-12-01T10:41:27.6608904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6609065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6609443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6609637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6609657Z 
2022-12-01T10:41:27.6609767Z Running tests...
2022-12-01T10:41:27.6610032Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6610347Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6610757Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6610781Z 
2022-12-01T10:41:27.6611048Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6611164Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6611184Z 
2022-12-01T10:41:27.6611274Z OK (skipped=1)
2022-12-01T10:41:27.6611293Z 
2022-12-01T10:41:27.6611419Z Generating XML reports...
2022-12-01T10:41:27.6611870Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102457.xml
2022-12-01T10:41:27.6612244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6612424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6612805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6612997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6613017Z 
2022-12-01T10:41:27.6613126Z Running tests...
2022-12-01T10:41:27.6613396Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6613740Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6614155Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.6614174Z 
2022-12-01T10:41:27.6614434Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6614547Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6614566Z 
2022-12-01T10:41:27.6614672Z OK (skipped=1)
2022-12-01T10:41:27.6614691Z 
2022-12-01T10:41:27.6614815Z Generating XML reports...
2022-12-01T10:41:27.6615260Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102459.xml
2022-12-01T10:41:27.6615630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6615810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6616173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6616363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6616383Z 
2022-12-01T10:41:27.6616492Z Running tests...
2022-12-01T10:41:27.6616756Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6617067Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6617354Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6617574Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11245
2022-12-01T10:41:27.6617794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11246
2022-12-01T10:41:27.6618206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6618384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6618766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6618958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6619327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6619502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6619881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6620072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6620322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6620560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6620967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6621366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6621599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6621829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6621930Z ok (3.966s)
2022-12-01T10:41:27.6621951Z 
2022-12-01T10:41:27.6622218Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6622332Z Ran 1 test in 3.966s
2022-12-01T10:41:27.6622351Z 
2022-12-01T10:41:27.6622430Z OK
2022-12-01T10:41:27.6622470Z 
2022-12-01T10:41:27.6622576Z Generating XML reports...
2022-12-01T10:41:27.6623069Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102501.xml
2022-12-01T10:41:27.6623452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6623630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6624009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6624201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6624221Z 
2022-12-01T10:41:27.6624333Z Running tests...
2022-12-01T10:41:27.6624600Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6624899Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6625183Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6625407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11577
2022-12-01T10:41:27.6625629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11578
2022-12-01T10:41:27.6626001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6626179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6626564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6626757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6627104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6627337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6627723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6627916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6628162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6628409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6628811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6629209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6629690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6629933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.6630161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6630402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.6630812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6631205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6631547Z STAGE:2022-12-01 10:25:10 11578:11578 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6631876Z STAGE:2022-12-01 10:25:10 11577:11577 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6632113Z [1669890312.740675] [7ff5451e42d4:11578:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6632439Z [1669890312.740675] [7ff5451e42d4:11578:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6632666Z [1669890312.742190] [7ff5451e42d4:11577:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6632899Z [1669890312.742190] [7ff5451e42d4:11577:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6633458Z STAGE:2022-12-01 10:25:13 11578:11578 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:13 11577:11577 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6633480Z 
2022-12-01T10:41:27.6633810Z STAGE:2022-12-01 10:25:13 11578:11578 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6634129Z STAGE:2022-12-01 10:25:13 11577:11577 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6634471Z STAGE:2022-12-01 10:25:13 11578:11578 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6634803Z STAGE:2022-12-01 10:25:13 11577:11577 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6634906Z ok (7.248s)
2022-12-01T10:41:27.6634926Z 
2022-12-01T10:41:27.6635322Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6635465Z Ran 1 test in 7.248s
2022-12-01T10:41:27.6635488Z 
2022-12-01T10:41:27.6635565Z OK
2022-12-01T10:41:27.6635583Z 
2022-12-01T10:41:27.6635711Z Generating XML reports...
2022-12-01T10:41:27.6636331Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102507.xml
2022-12-01T10:41:27.6636711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6636975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6637548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6637799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6637821Z 
2022-12-01T10:41:27.6637931Z Running tests...
2022-12-01T10:41:27.6638200Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6638494Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6638771Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6638992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11938
2022-12-01T10:41:27.6639211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11939
2022-12-01T10:41:27.6639585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6639770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6640153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6640345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6640688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6640867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6641239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6641426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6641673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6641924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6642380Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6642790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6643022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6643244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.6643476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6643722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.6644119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6644522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6644862Z STAGE:2022-12-01 10:25:20 11939:11939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6645196Z STAGE:2022-12-01 10:25:20 11938:11938 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6645434Z [1669890322.148421] [7ff5451e42d4:11939:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6645675Z [1669890322.148421] [7ff5451e42d4:11939:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6645883Z [1669890322.136838] [7ff5451e42d4:11938:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6646123Z [1669890322.136838] [7ff5451e42d4:11938:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6646739Z STAGE:2022-12-01 10:25:22 11939:11939 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:22 11938:11938 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6646760Z 
2022-12-01T10:41:27.6647090Z STAGE:2022-12-01 10:25:22 11938:11938 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6647412Z STAGE:2022-12-01 10:25:22 11939:11939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6647748Z STAGE:2022-12-01 10:25:22 11938:11938 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6648082Z STAGE:2022-12-01 10:25:22 11939:11939 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6648184Z ok (7.280s)
2022-12-01T10:41:27.6648204Z 
2022-12-01T10:41:27.6648472Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6648591Z Ran 1 test in 7.281s
2022-12-01T10:41:27.6648610Z 
2022-12-01T10:41:27.6648684Z OK
2022-12-01T10:41:27.6648706Z 
2022-12-01T10:41:27.6648832Z Generating XML reports...
2022-12-01T10:41:27.6649284Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102516.xml
2022-12-01T10:41:27.6649658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6649852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6650233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6650428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6650447Z 
2022-12-01T10:41:27.6650557Z Running tests...
2022-12-01T10:41:27.6650807Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6651127Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6651485Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6651714Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12395
2022-12-01T10:41:27.6651937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12396
2022-12-01T10:41:27.6652308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6652486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6652865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6653058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6653406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6653591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6653970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6654159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6654405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6654651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6655048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6655448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6655680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6655961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.6656189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6656508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.6656916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6657379Z STAGE:2022-12-01 10:25:29 12396:12396 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6657863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6658221Z STAGE:2022-12-01 10:25:29 12395:12395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6658557Z [1669890331.477988] [7ff5451e42d4:12396:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6658826Z [1669890331.477988] [7ff5451e42d4:12396:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6659036Z [1669890331.522367] [7ff5451e42d4:12395:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6659266Z [1669890331.522367] [7ff5451e42d4:12395:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6659822Z STAGE:2022-12-01 10:25:31 12396:12396 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:31 12395:12395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6659844Z 
2022-12-01T10:41:27.6660172Z STAGE:2022-12-01 10:25:32 12396:12396 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6660564Z STAGE:2022-12-01 10:25:32 12395:12395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6660916Z STAGE:2022-12-01 10:25:32 12396:12396 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6661254Z STAGE:2022-12-01 10:25:32 12395:12395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6661360Z ok (7.124s)
2022-12-01T10:41:27.6661380Z 
2022-12-01T10:41:27.6661649Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6661763Z Ran 1 test in 7.124s
2022-12-01T10:41:27.6661783Z 
2022-12-01T10:41:27.6661858Z OK
2022-12-01T10:41:27.6661877Z 
2022-12-01T10:41:27.6662001Z Generating XML reports...
2022-12-01T10:41:27.6662452Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102526.xml
2022-12-01T10:41:27.6662827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6663013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6663398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6663594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6663614Z 
2022-12-01T10:41:27.6663727Z Running tests...
2022-12-01T10:41:27.6663976Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6664297Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6664567Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6664789Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12787
2022-12-01T10:41:27.6665008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12788
2022-12-01T10:41:27.6665448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6665627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6666007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6666200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6666547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6666724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6667104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6667293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6667545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6667794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6668198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6668595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6668828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6669053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.6669280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6669715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.6670206Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6670617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6670955Z STAGE:2022-12-01 10:25:39 12787:12787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6671281Z STAGE:2022-12-01 10:25:39 12788:12788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6671520Z [1669890341.069470] [7ff5451e42d4:12788:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6671763Z [1669890341.069470] [7ff5451e42d4:12788:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6671976Z [1669890341.075998] [7ff5451e42d4:12787:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6672217Z [1669890341.075998] [7ff5451e42d4:12787:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6672772Z STAGE:2022-12-01 10:25:41 12788:12788 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:41 12787:12787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6672793Z 
2022-12-01T10:41:27.6673120Z STAGE:2022-12-01 10:25:41 12788:12788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6673442Z STAGE:2022-12-01 10:25:41 12787:12787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6673778Z STAGE:2022-12-01 10:25:41 12788:12788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6674105Z STAGE:2022-12-01 10:25:41 12787:12787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6674208Z ok (7.214s)
2022-12-01T10:41:27.6674286Z 
2022-12-01T10:41:27.6674569Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6674687Z Ran 1 test in 7.214s
2022-12-01T10:41:27.6674707Z 
2022-12-01T10:41:27.6674781Z OK
2022-12-01T10:41:27.6674801Z 
2022-12-01T10:41:27.6674926Z Generating XML reports...
2022-12-01T10:41:27.6675378Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102535.xml
2022-12-01T10:41:27.6675750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6675927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6676306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6676499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6676519Z 
2022-12-01T10:41:27.6676631Z Running tests...
2022-12-01T10:41:27.6676884Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6677202Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6677471Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6677692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13244
2022-12-01T10:41:27.6677911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13245
2022-12-01T10:41:27.6678286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6678463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6678844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6679035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6679439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6679623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6680009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6680201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6680446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6680738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6681145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6681544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6681765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6681995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6682156Z skip: Skipped due to small world size. (3.977s)
2022-12-01T10:41:27.6682177Z 
2022-12-01T10:41:27.6682447Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6682560Z Ran 1 test in 3.977s
2022-12-01T10:41:27.6682581Z 
2022-12-01T10:41:27.6682689Z OK (skipped=1)
2022-12-01T10:41:27.6682708Z 
2022-12-01T10:41:27.6682833Z Generating XML reports...
2022-12-01T10:41:27.6683284Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102544.xml
2022-12-01T10:41:27.6683660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6683899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6684287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6684478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6684498Z 
2022-12-01T10:41:27.6684609Z Running tests...
2022-12-01T10:41:27.6684876Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6685191Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6685456Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6685678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13566
2022-12-01T10:41:27.6685878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13567
2022-12-01T10:41:27.6686258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6686435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6686821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6687014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6687379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6687552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6687929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6688117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6688344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6688640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6689053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6689452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6689681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6689908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6690070Z skip: Skipped due to small world size. (3.919s)
2022-12-01T10:41:27.6690090Z 
2022-12-01T10:41:27.6690356Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6690469Z Ran 1 test in 3.919s
2022-12-01T10:41:27.6690489Z 
2022-12-01T10:41:27.6690582Z OK (skipped=1)
2022-12-01T10:41:27.6690601Z 
2022-12-01T10:41:27.6690729Z Generating XML reports...
2022-12-01T10:41:27.6691183Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102551.xml
2022-12-01T10:41:27.6691560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6691737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6692117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6692309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6692329Z 
2022-12-01T10:41:27.6692440Z Running tests...
2022-12-01T10:41:27.6692687Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6693005Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6693328Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6693550Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13871
2022-12-01T10:41:27.6693770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13872
2022-12-01T10:41:27.6694144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6694321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6694701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6694895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6695243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6695421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6695804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6695993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6696242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6696489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6696890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6697292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6697524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6697796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6697968Z skip: Skipped due to small world size. (4.013s)
2022-12-01T10:41:27.6697988Z 
2022-12-01T10:41:27.6698257Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6698370Z Ran 1 test in 4.013s
2022-12-01T10:41:27.6698390Z 
2022-12-01T10:41:27.6698498Z OK (skipped=1)
2022-12-01T10:41:27.6698517Z 
2022-12-01T10:41:27.6698643Z Generating XML reports...
2022-12-01T10:41:27.6699093Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102557.xml
2022-12-01T10:41:27.6699466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6699626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6700010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6700210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6700230Z 
2022-12-01T10:41:27.6700345Z Running tests...
2022-12-01T10:41:27.6700612Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6700926Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6701192Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6701413Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14181
2022-12-01T10:41:27.6701632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14182
2022-12-01T10:41:27.6701988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6702170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6702610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6702800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6703165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6703341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6703715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6703904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6704131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6704376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6704786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6705186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6705416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6705645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6705806Z skip: Skipped due to small world size. (4.055s)
2022-12-01T10:41:27.6705827Z 
2022-12-01T10:41:27.6706095Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6706207Z Ran 1 test in 4.055s
2022-12-01T10:41:27.6706227Z 
2022-12-01T10:41:27.6706316Z OK (skipped=1)
2022-12-01T10:41:27.6706356Z 
2022-12-01T10:41:27.6706462Z Generating XML reports...
2022-12-01T10:41:27.6706911Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102603.xml
2022-12-01T10:41:27.6707332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6707516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6707902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6708094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6708114Z 
2022-12-01T10:41:27.6708227Z Running tests...
2022-12-01T10:41:27.6708495Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6708789Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6709042Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6709266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14497
2022-12-01T10:41:27.6709722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14498
2022-12-01T10:41:27.6710105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6710285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6710668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6710861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6711208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6711384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6711758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6712030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6712277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6712521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6712926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6713322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6713555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6713876Z STAGE:2022-12-01 10:26:13 14498:14498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6714101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6714440Z STAGE:2022-12-01 10:26:13 14497:14497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6714682Z [1669890375.044114] [7ff5451e42d4:14498:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6714926Z [1669890375.044114] [7ff5451e42d4:14498:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6715157Z [1669890375.118929] [7ff5451e42d4:14497:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6715393Z [1669890375.118929] [7ff5451e42d4:14497:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6715950Z STAGE:2022-12-01 10:26:15 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:15 14497:14497 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6715975Z 
2022-12-01T10:41:27.6716378Z STAGE:2022-12-01 10:26:15 14498:14498 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6716716Z STAGE:2022-12-01 10:26:15 14497:14497 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6717033Z STAGE:2022-12-01 10:26:15 14498:14498 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6717362Z STAGE:2022-12-01 10:26:15 14497:14497 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6717465Z ok (7.144s)
2022-12-01T10:41:27.6717485Z 
2022-12-01T10:41:27.6717749Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6717863Z Ran 1 test in 7.145s
2022-12-01T10:41:27.6717883Z 
2022-12-01T10:41:27.6717975Z OK
2022-12-01T10:41:27.6717994Z 
2022-12-01T10:41:27.6718119Z Generating XML reports...
2022-12-01T10:41:27.6718570Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102609.xml
2022-12-01T10:41:27.6718953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6719113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6719493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6719686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6719705Z 
2022-12-01T10:41:27.6719814Z Running tests...
2022-12-01T10:41:27.6720079Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6720395Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6720649Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6720874Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14903
2022-12-01T10:41:27.6721128Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14904
2022-12-01T10:41:27.6721509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6721686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6722069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6722265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6722629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6722806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6723182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6723380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6723610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6723857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6724261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6724657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6724889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6725225Z STAGE:2022-12-01 10:26:22 14904:14904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6725450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6725825Z STAGE:2022-12-01 10:26:22 14903:14903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6726071Z [1669890384.289104] [7ff5451e42d4:14903:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6726292Z [1669890384.289104] [7ff5451e42d4:14903:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6726527Z [1669890384.339642] [7ff5451e42d4:14904:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6726764Z [1669890384.339642] [7ff5451e42d4:14904:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6727318Z STAGE:2022-12-01 10:26:24 14903:14903 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:24 14904:14904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6727342Z 
2022-12-01T10:41:27.6727673Z STAGE:2022-12-01 10:26:24 14904:14904 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6727999Z STAGE:2022-12-01 10:26:24 14903:14903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6728334Z STAGE:2022-12-01 10:26:24 14904:14904 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6728664Z STAGE:2022-12-01 10:26:24 14903:14903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6728769Z ok (7.128s)
2022-12-01T10:41:27.6728789Z 
2022-12-01T10:41:27.6729057Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6729153Z Ran 1 test in 7.128s
2022-12-01T10:41:27.6729173Z 
2022-12-01T10:41:27.6729267Z OK
2022-12-01T10:41:27.6729286Z 
2022-12-01T10:41:27.6729411Z Generating XML reports...
2022-12-01T10:41:27.6729864Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102618.xml
2022-12-01T10:41:27.6730298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6730475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6730859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6731054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6731074Z 
2022-12-01T10:41:27.6731165Z Running tests...
2022-12-01T10:41:27.6731432Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6731746Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6732015Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: UCC all_reduce multigpu skipped (0.002s)
2022-12-01T10:41:27.6732039Z 
2022-12-01T10:41:27.6732304Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6732419Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6732439Z 
2022-12-01T10:41:27.6732549Z OK (skipped=1)
2022-12-01T10:41:27.6732568Z 
2022-12-01T10:41:27.6732692Z Generating XML reports...
2022-12-01T10:41:27.6733145Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102628.xml
2022-12-01T10:41:27.6733501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6733679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6734060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6734254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6734273Z 
2022-12-01T10:41:27.6734383Z Running tests...
2022-12-01T10:41:27.6734654Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6735012Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6735300Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: UCC all_reduce multigpu skipped (0.002s)
2022-12-01T10:41:27.6735320Z 
2022-12-01T10:41:27.6735588Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6735683Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6735702Z 
2022-12-01T10:41:27.6735811Z OK (skipped=1)
2022-12-01T10:41:27.6735830Z 
2022-12-01T10:41:27.6735953Z Generating XML reports...
2022-12-01T10:41:27.6736402Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102630.xml
2022-12-01T10:41:27.6736778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6736960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6737343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6737537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6737557Z 
2022-12-01T10:41:27.6737668Z Running tests...
2022-12-01T10:41:27.6737915Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6738231Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6738493Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6738771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15528
2022-12-01T10:41:27.6738993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15529
2022-12-01T10:41:27.6739368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6739609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6739999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6740174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6740543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6740718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6741093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6741283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6741533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6741786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6742192Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6742593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6742807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6743145Z STAGE:2022-12-01 10:26:36 15528:15528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6743371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6743701Z STAGE:2022-12-01 10:26:36 15529:15529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6743939Z [1669890398.032712] [7ff5451e42d4:15528:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6744230Z [1669890398.032712] [7ff5451e42d4:15528:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6744471Z [1669890397.989527] [7ff5451e42d4:15529:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6744708Z [1669890397.989527] [7ff5451e42d4:15529:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6745266Z STAGE:2022-12-01 10:26:38 15528:15528 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:38 15529:15529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6745287Z 
2022-12-01T10:41:27.6745615Z STAGE:2022-12-01 10:26:38 15528:15528 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6745919Z STAGE:2022-12-01 10:26:38 15529:15529 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6746261Z STAGE:2022-12-01 10:26:38 15528:15528 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6746593Z STAGE:2022-12-01 10:26:38 15529:15529 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6746697Z ok (7.097s)
2022-12-01T10:41:27.6746717Z 
2022-12-01T10:41:27.6746983Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6747097Z Ran 1 test in 7.097s
2022-12-01T10:41:27.6747117Z 
2022-12-01T10:41:27.6747211Z OK
2022-12-01T10:41:27.6747230Z 
2022-12-01T10:41:27.6747355Z Generating XML reports...
2022-12-01T10:41:27.6747783Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102632.xml
2022-12-01T10:41:27.6748159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6748388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6748774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6748967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6748987Z 
2022-12-01T10:41:27.6749098Z Running tests...
2022-12-01T10:41:27.6749580Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6749912Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6750323Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-12-01T10:41:27.6750344Z 
2022-12-01T10:41:27.6750588Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6750704Z Ran 1 test in 0.003s
2022-12-01T10:41:27.6750724Z 
2022-12-01T10:41:27.6750837Z OK (skipped=1)
2022-12-01T10:41:27.6750856Z 
2022-12-01T10:41:27.6750980Z Generating XML reports...
2022-12-01T10:41:27.6751429Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102641.xml
2022-12-01T10:41:27.6751809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6751987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6752369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6752564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6752584Z 
2022-12-01T10:41:27.6752673Z Running tests...
2022-12-01T10:41:27.6752942Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6753253Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6753581Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6753815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16064
2022-12-01T10:41:27.6754033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16065
2022-12-01T10:41:27.6754409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6754585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6754945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6755141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6755504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6755685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6756064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6756257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6756506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6756750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6757151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6757535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6757770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6758183Z STAGE:2022-12-01 10:26:47 16064:16064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6758414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6758747Z STAGE:2022-12-01 10:26:47 16065:16065 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6758984Z [1669890409.344682] [7ff5451e42d4:16064:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6759226Z [1669890409.344682] [7ff5451e42d4:16064:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6759457Z [1669890409.364850] [7ff5451e42d4:16065:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6759695Z [1669890409.364850] [7ff5451e42d4:16065:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6760237Z STAGE:2022-12-01 10:26:49 16064:16064 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:49 16065:16065 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6760276Z 
2022-12-01T10:41:27.6760589Z STAGE:2022-12-01 10:26:49 16065:16065 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6760913Z STAGE:2022-12-01 10:26:49 16064:16064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6761249Z STAGE:2022-12-01 10:26:49 16065:16065 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6761578Z STAGE:2022-12-01 10:26:49 16064:16064 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6761683Z ok (7.073s)
2022-12-01T10:41:27.6761701Z 
2022-12-01T10:41:27.6761968Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6762082Z Ran 1 test in 7.073s
2022-12-01T10:41:27.6762102Z 
2022-12-01T10:41:27.6762199Z OK
2022-12-01T10:41:27.6762218Z 
2022-12-01T10:41:27.6762345Z Generating XML reports...
2022-12-01T10:41:27.6762837Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102643.xml
2022-12-01T10:41:27.6763226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6763401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6763785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6763979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6763998Z 
2022-12-01T10:41:27.6764109Z Running tests...
2022-12-01T10:41:27.6764377Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6764690Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6764939Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6765165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16485
2022-12-01T10:41:27.6765384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16486
2022-12-01T10:41:27.6765758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6765934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6766316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6766506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6766871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6767099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6767464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6767652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6767903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6768148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6768548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6768949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6769183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6769524Z STAGE:2022-12-01 10:26:56 16485:16485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6769733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6770062Z STAGE:2022-12-01 10:26:57 16486:16486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6770301Z [1669890418.817456] [7ff5451e42d4:16486:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6770544Z [1669890418.817456] [7ff5451e42d4:16486:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6770776Z [1669890418.802456] [7ff5451e42d4:16485:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6771015Z [1669890418.802456] [7ff5451e42d4:16485:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6771620Z STAGE:2022-12-01 10:26:59 16486:16486 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:59 16485:16485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6771642Z 
2022-12-01T10:41:27.6771980Z STAGE:2022-12-01 10:26:59 16486:16486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6772304Z STAGE:2022-12-01 10:26:59 16485:16485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6772849Z STAGE:2022-12-01 10:26:59 16486:16486 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:59 16485:16485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6772870Z 
2022-12-01T10:41:27.6772973Z ok (7.317s)
2022-12-01T10:41:27.6772993Z 
2022-12-01T10:41:27.6773240Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6773356Z Ran 1 test in 7.317s
2022-12-01T10:41:27.6773376Z 
2022-12-01T10:41:27.6773473Z OK
2022-12-01T10:41:27.6773493Z 
2022-12-01T10:41:27.6773617Z Generating XML reports...
2022-12-01T10:41:27.6774076Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102653.xml
2022-12-01T10:41:27.6774453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6774629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6775011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6775208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6775229Z 
2022-12-01T10:41:27.6775319Z Running tests...
2022-12-01T10:41:27.6775588Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6775902Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6776230Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6776450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16902
2022-12-01T10:41:27.6776669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16903
2022-12-01T10:41:27.6777046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6777223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6777586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6777783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6778146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6778325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6778702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6778893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6779143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6779390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6779794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6780172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6780403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6780792Z STAGE:2022-12-01 10:27:06 16903:16903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6781066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6781407Z STAGE:2022-12-01 10:27:06 16902:16902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6781644Z [1669890428.263448] [7ff5451e42d4:16902:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6781887Z [1669890428.263448] [7ff5451e42d4:16902:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6782119Z [1669890428.305997] [7ff5451e42d4:16903:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6782356Z [1669890428.305997] [7ff5451e42d4:16903:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6782912Z STAGE:2022-12-01 10:27:08 16902:16902 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:27:08 16903:16903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6782933Z 
2022-12-01T10:41:27.6783242Z STAGE:2022-12-01 10:27:08 16902:16902 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6783566Z STAGE:2022-12-01 10:27:08 16903:16903 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6783901Z STAGE:2022-12-01 10:27:08 16902:16902 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6784231Z STAGE:2022-12-01 10:27:08 16903:16903 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6784334Z ok (7.245s)
2022-12-01T10:41:27.6784357Z 
2022-12-01T10:41:27.6784626Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6784742Z Ran 1 test in 7.245s
2022-12-01T10:41:27.6784763Z 
2022-12-01T10:41:27.6784906Z OK
2022-12-01T10:41:27.6784926Z 
2022-12-01T10:41:27.6785054Z Generating XML reports...
2022-12-01T10:41:27.6785491Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102702.xml
2022-12-01T10:41:27.6785867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6786045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6786424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6786617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6786637Z 
2022-12-01T10:41:27.6786746Z Running tests...
2022-12-01T10:41:27.6787012Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6787324Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6787612Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-12-01T10:41:27.6787656Z 
2022-12-01T10:41:27.6787903Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6788017Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6788039Z 
2022-12-01T10:41:27.6788152Z OK (skipped=1)
2022-12-01T10:41:27.6788175Z 
2022-12-01T10:41:27.6788298Z Generating XML reports...
2022-12-01T10:41:27.6788750Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102712.xml
2022-12-01T10:41:27.6789122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6789305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6789929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6790196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6790240Z 
2022-12-01T10:41:27.6790343Z Running tests...
2022-12-01T10:41:27.6790620Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6790933Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6791238Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.004s)
2022-12-01T10:41:27.6791260Z 
2022-12-01T10:41:27.6791523Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6791639Z Ran 1 test in 0.004s
2022-12-01T10:41:27.6791659Z 
2022-12-01T10:41:27.6791770Z OK (skipped=1)
2022-12-01T10:41:27.6791792Z 
2022-12-01T10:41:27.6791917Z Generating XML reports...
2022-12-01T10:41:27.6792348Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102714.xml
2022-12-01T10:41:27.6792731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6792908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6793289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6793483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6793507Z 
2022-12-01T10:41:27.6793617Z Running tests...
2022-12-01T10:41:27.6793879Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6794195Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6794507Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-12-01T10:41:27.6794621Z 
2022-12-01T10:41:27.6794879Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6795001Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6795023Z 
2022-12-01T10:41:27.6795132Z OK (skipped=1)
2022-12-01T10:41:27.6795153Z 
2022-12-01T10:41:27.6795277Z Generating XML reports...
2022-12-01T10:41:27.6795725Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102716.xml
2022-12-01T10:41:27.6796097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6796272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6796654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6796848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6796874Z 
2022-12-01T10:41:27.6796964Z Running tests...
2022-12-01T10:41:27.6797235Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6797549Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6797794Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:41:27.6797815Z 
2022-12-01T10:41:27.6798076Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6798191Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6798212Z 
2022-12-01T10:41:27.6798321Z OK (skipped=1)
2022-12-01T10:41:27.6798343Z 
2022-12-01T10:41:27.6798468Z Generating XML reports...
2022-12-01T10:41:27.6798893Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102718.xml
2022-12-01T10:41:27.6799266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6799493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6799885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6800078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6800099Z 
2022-12-01T10:41:27.6800208Z Running tests...
2022-12-01T10:41:27.6800475Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6800787Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6801046Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:41:27.6801068Z 
2022-12-01T10:41:27.6801311Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6801427Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6801447Z 
2022-12-01T10:41:27.6801559Z OK (skipped=1)
2022-12-01T10:41:27.6801579Z 
2022-12-01T10:41:27.6801709Z Generating XML reports...
2022-12-01T10:41:27.6802158Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102720.xml
2022-12-01T10:41:27.6802535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6802713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6803091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6803282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6803302Z 
2022-12-01T10:41:27.6803393Z Running tests...
2022-12-01T10:41:27.6803661Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6803974Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6804291Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:41:27.6804312Z 
2022-12-01T10:41:27.6804578Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6804691Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6804711Z 
2022-12-01T10:41:27.6804818Z OK (skipped=1)
2022-12-01T10:41:27.6804841Z 
2022-12-01T10:41:27.6804967Z Generating XML reports...
2022-12-01T10:41:27.6805397Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102722.xml
2022-12-01T10:41:27.6805769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6805944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6806328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6806531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6806553Z 
2022-12-01T10:41:27.6806661Z Running tests...
2022-12-01T10:41:27.6806932Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6807249Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6807525Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:41:27.6807547Z 
2022-12-01T10:41:27.6807791Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6807907Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6807929Z 
2022-12-01T10:41:27.6808039Z OK (skipped=1)
2022-12-01T10:41:27.6808059Z 
2022-12-01T10:41:27.6808184Z Generating XML reports...
2022-12-01T10:41:27.6808631Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102724.xml
2022-12-01T10:41:27.6809054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6809242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6809623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6809816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6809838Z 
2022-12-01T10:41:27.6809928Z Running tests...
2022-12-01T10:41:27.6810196Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6810511Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6810773Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:41:27.6810794Z 
2022-12-01T10:41:27.6811070Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6811189Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6811210Z 
2022-12-01T10:41:27.6811319Z OK (skipped=1)
2022-12-01T10:41:27.6811340Z 
2022-12-01T10:41:27.6811468Z Generating XML reports...
2022-12-01T10:41:27.6811914Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102726.xml
2022-12-01T10:41:27.6812270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6812446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6812827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6813024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6813045Z 
2022-12-01T10:41:27.6813155Z Running tests...
2022-12-01T10:41:27.6813475Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6813793Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6814070Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:41:27.6814091Z 
2022-12-01T10:41:27.6814353Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6814448Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6814469Z 
2022-12-01T10:41:27.6814578Z OK (skipped=1)
2022-12-01T10:41:27.6814597Z 
2022-12-01T10:41:27.6814720Z Generating XML reports...
2022-12-01T10:41:27.6815167Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102728.xml
2022-12-01T10:41:27.6815545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6815726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6816110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6816307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6816326Z 
2022-12-01T10:41:27.6816418Z Running tests...
2022-12-01T10:41:27.6816682Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6816999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6817256Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:41:27.6817278Z 
2022-12-01T10:41:27.6817541Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6817655Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6817675Z 
2022-12-01T10:41:27.6817783Z OK (skipped=1)
2022-12-01T10:41:27.6817806Z 
2022-12-01T10:41:27.6817932Z Generating XML reports...
2022-12-01T10:41:27.6818421Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102730.xml
2022-12-01T10:41:27.6818785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6818964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6819343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6819536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6819559Z 
2022-12-01T10:41:27.6819669Z Running tests...
2022-12-01T10:41:27.6819937Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6820246Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6820536Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6820558Z 
2022-12-01T10:41:27.6820819Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6820917Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6820940Z 
2022-12-01T10:41:27.6821049Z OK (skipped=1)
2022-12-01T10:41:27.6821072Z 
2022-12-01T10:41:27.6821197Z Generating XML reports...
2022-12-01T10:41:27.6821643Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102732.xml
2022-12-01T10:41:27.6822019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6822199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6822580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6822824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6822847Z 
2022-12-01T10:41:27.6822940Z Running tests...
2022-12-01T10:41:27.6823208Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6823518Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6823803Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6823826Z 
2022-12-01T10:41:27.6824092Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6824207Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6824230Z 
2022-12-01T10:41:27.6824343Z OK (skipped=1)
2022-12-01T10:41:27.6824365Z 
2022-12-01T10:41:27.6824489Z Generating XML reports...
2022-12-01T10:41:27.6824936Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102735.xml
2022-12-01T10:41:27.6825302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6825481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6825867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6826062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6826084Z 
2022-12-01T10:41:27.6826194Z Running tests...
2022-12-01T10:41:27.6826455Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6826772Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6827082Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6827103Z 
2022-12-01T10:41:27.6827372Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6827542Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6827564Z 
2022-12-01T10:41:27.6827679Z OK (skipped=1)
2022-12-01T10:41:27.6827704Z 
2022-12-01T10:41:27.6827830Z Generating XML reports...
2022-12-01T10:41:27.6828277Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102737.xml
2022-12-01T10:41:27.6828649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6828808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6829192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6829560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6829583Z 
2022-12-01T10:41:27.6829699Z Running tests...
2022-12-01T10:41:27.6829977Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6830294Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6830589Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6830612Z 
2022-12-01T10:41:27.6830874Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6830989Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6831009Z 
2022-12-01T10:41:27.6831099Z OK (skipped=1)
2022-12-01T10:41:27.6831119Z 
2022-12-01T10:41:27.6831243Z Generating XML reports...
2022-12-01T10:41:27.6831687Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102739.xml
2022-12-01T10:41:27.6832059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6832323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6832709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6832900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6832923Z 
2022-12-01T10:41:27.6833030Z Running tests...
2022-12-01T10:41:27.6833277Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6833588Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6833893Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6833913Z 
2022-12-01T10:41:27.6834173Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6834288Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6834309Z 
2022-12-01T10:41:27.6834420Z OK (skipped=1)
2022-12-01T10:41:27.6834440Z 
2022-12-01T10:41:27.6834564Z Generating XML reports...
2022-12-01T10:41:27.6835009Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102741.xml
2022-12-01T10:41:27.6835380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6835537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6835917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6836108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6836130Z 
2022-12-01T10:41:27.6836240Z Running tests...
2022-12-01T10:41:27.6836505Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6836817Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6837175Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6837198Z 
2022-12-01T10:41:27.6837471Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6837584Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6837606Z 
2022-12-01T10:41:27.6837697Z OK (skipped=1)
2022-12-01T10:41:27.6837735Z 
2022-12-01T10:41:27.6837843Z Generating XML reports...
2022-12-01T10:41:27.6838292Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102743.xml
2022-12-01T10:41:27.6838663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6838840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6839220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6839424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6839446Z 
2022-12-01T10:41:27.6839553Z Running tests...
2022-12-01T10:41:27.6839820Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6840114Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6840422Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6840443Z 
2022-12-01T10:41:27.6840703Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6840815Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6840835Z 
2022-12-01T10:41:27.6840946Z OK (skipped=1)
2022-12-01T10:41:27.6840966Z 
2022-12-01T10:41:27.6841090Z Generating XML reports...
2022-12-01T10:41:27.6841536Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102745.xml
2022-12-01T10:41:27.6841968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6842145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6842508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6842700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6842721Z 
2022-12-01T10:41:27.6842830Z Running tests...
2022-12-01T10:41:27.6843089Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6843398Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6843691Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6843717Z 
2022-12-01T10:41:27.6843986Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6844100Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6844120Z 
2022-12-01T10:41:27.6844233Z OK (skipped=1)
2022-12-01T10:41:27.6844253Z 
2022-12-01T10:41:27.6844360Z Generating XML reports...
2022-12-01T10:41:27.6844806Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102747.xml
2022-12-01T10:41:27.6845180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6845355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6845730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6845923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6845949Z 
2022-12-01T10:41:27.6846058Z Running tests...
2022-12-01T10:41:27.6846365Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6846671Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6846975Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6846997Z 
2022-12-01T10:41:27.6847260Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6847371Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6847391Z 
2022-12-01T10:41:27.6847499Z OK (skipped=1)
2022-12-01T10:41:27.6847519Z 
2022-12-01T10:41:27.6847642Z Generating XML reports...
2022-12-01T10:41:27.6848084Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102750.xml
2022-12-01T10:41:27.6848455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6848640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6849003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6849198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6849218Z 
2022-12-01T10:41:27.6849326Z Running tests...
2022-12-01T10:41:27.6849589Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6849897Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6850185Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6850205Z 
2022-12-01T10:41:27.6850468Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6850632Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6850651Z 
2022-12-01T10:41:27.6850757Z OK (skipped=1)
2022-12-01T10:41:27.6850779Z 
2022-12-01T10:41:27.6850884Z Generating XML reports...
2022-12-01T10:41:27.6851329Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102752.xml
2022-12-01T10:41:27.6851698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6851873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6852250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6852442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6852461Z 
2022-12-01T10:41:27.6852571Z Running tests...
2022-12-01T10:41:27.6852835Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6853150Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6853434Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6853455Z 
2022-12-01T10:41:27.6853715Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6853827Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6853846Z 
2022-12-01T10:41:27.6853954Z OK (skipped=1)
2022-12-01T10:41:27.6853973Z 
2022-12-01T10:41:27.6854092Z Generating XML reports...
2022-12-01T10:41:27.6854538Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102754.xml
2022-12-01T10:41:27.6854912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6855089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6855521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6855703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6855724Z 
2022-12-01T10:41:27.6855833Z Running tests...
2022-12-01T10:41:27.6856102Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6856416Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6856714Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6856736Z 
2022-12-01T10:41:27.6856995Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6857108Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6857130Z 
2022-12-01T10:41:27.6857238Z OK (skipped=1)
2022-12-01T10:41:27.6857259Z 
2022-12-01T10:41:27.6857365Z Generating XML reports...
2022-12-01T10:41:27.6857819Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102756.xml
2022-12-01T10:41:27.6858194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6858370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6858749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6858944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6858965Z 
2022-12-01T10:41:27.6859073Z Running tests...
2022-12-01T10:41:27.6859344Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6859654Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6859944Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6860033Z 
2022-12-01T10:41:27.6860283Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6860397Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6860419Z 
2022-12-01T10:41:27.6860526Z OK (skipped=1)
2022-12-01T10:41:27.6860546Z 
2022-12-01T10:41:27.6860671Z Generating XML reports...
2022-12-01T10:41:27.6861117Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102758.xml
2022-12-01T10:41:27.6861490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6861670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6862051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6862234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6862257Z 
2022-12-01T10:41:27.6862372Z Running tests...
2022-12-01T10:41:27.6862639Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6862949Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6863253Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6863275Z 
2022-12-01T10:41:27.6863541Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6863657Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6863679Z 
2022-12-01T10:41:27.6863787Z OK (skipped=1)
2022-12-01T10:41:27.6863809Z 
2022-12-01T10:41:27.6863934Z Generating XML reports...
2022-12-01T10:41:27.6864359Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102800.xml
2022-12-01T10:41:27.6864783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6864969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6865357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6865554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6865576Z 
2022-12-01T10:41:27.6865688Z Running tests...
2022-12-01T10:41:27.6865956Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6866271Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6866569Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6866609Z 
2022-12-01T10:41:27.6866863Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6866980Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6867003Z 
2022-12-01T10:41:27.6867112Z OK (skipped=1)
2022-12-01T10:41:27.6867135Z 
2022-12-01T10:41:27.6867261Z Generating XML reports...
2022-12-01T10:41:27.6867706Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102802.xml
2022-12-01T10:41:27.6868075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6868249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6868629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6868803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6868841Z 
2022-12-01T10:41:27.6868930Z Running tests...
2022-12-01T10:41:27.6869245Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6869771Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6870070Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:41:27.6870091Z 
2022-12-01T10:41:27.6870355Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6870468Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6870487Z 
2022-12-01T10:41:27.6870595Z OK (skipped=1)
2022-12-01T10:41:27.6870615Z 
2022-12-01T10:41:27.6870739Z Generating XML reports...
2022-12-01T10:41:27.6871166Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102804.xml
2022-12-01T10:41:27.6871542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6871722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6872109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6872302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6872322Z 
2022-12-01T10:41:27.6872430Z Running tests...
2022-12-01T10:41:27.6872692Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6873003Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6873305Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:41:27.6873325Z 
2022-12-01T10:41:27.6873570Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6873685Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6873706Z 
2022-12-01T10:41:27.6873814Z OK (skipped=1)
2022-12-01T10:41:27.6873833Z 
2022-12-01T10:41:27.6874021Z Generating XML reports...
2022-12-01T10:41:27.6874480Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102806.xml
2022-12-01T10:41:27.6874852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6875026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6875401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6875592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6875613Z 
2022-12-01T10:41:27.6875703Z Running tests...
2022-12-01T10:41:27.6875969Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6876277Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6876550Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6876775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19880
2022-12-01T10:41:27.6876992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19881
2022-12-01T10:41:27.6877364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6877544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6877908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6878101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6878464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6878704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6879085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6879273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6879522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6879768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6880173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6880558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6880838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6881073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6881309Z [1669890494.897484] [7ff5451e42d4:19880:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6881554Z [1669890494.897484] [7ff5451e42d4:19880:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6881780Z [1669890494.902385] [7ff5451e42d4:19881:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6882010Z [1669890494.902385] [7ff5451e42d4:19881:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6882253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.6882494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.6882932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6883346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.6883451Z ok (7.181s)
2022-12-01T10:41:27.6883471Z 
2022-12-01T10:41:27.6883737Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6883848Z Ran 1 test in 7.181s
2022-12-01T10:41:27.6883867Z 
2022-12-01T10:41:27.6883960Z OK
2022-12-01T10:41:27.6883979Z 
2022-12-01T10:41:27.6884102Z Generating XML reports...
2022-12-01T10:41:27.6884548Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102808.xml
2022-12-01T10:41:27.6884923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6885082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6885474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6885669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6885688Z 
2022-12-01T10:41:27.6885798Z Running tests...
2022-12-01T10:41:27.6886061Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6886373Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6886774Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.6886794Z 
2022-12-01T10:41:27.6887059Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6887167Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6887187Z 
2022-12-01T10:41:27.6887276Z OK (skipped=1)
2022-12-01T10:41:27.6887341Z 
2022-12-01T10:41:27.6887472Z Generating XML reports...
2022-12-01T10:41:27.6887923Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102818.xml
2022-12-01T10:41:27.6888295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6888471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6888847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6889039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6889058Z 
2022-12-01T10:41:27.6889166Z Running tests...
2022-12-01T10:41:27.6889411Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6889724Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6890117Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.6890137Z 
2022-12-01T10:41:27.6890395Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6890509Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6890528Z 
2022-12-01T10:41:27.6890635Z OK (skipped=1)
2022-12-01T10:41:27.6890654Z 
2022-12-01T10:41:27.6890780Z Generating XML reports...
2022-12-01T10:41:27.6891225Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102820.xml
2022-12-01T10:41:27.6891599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6891759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6892135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6892330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6892405Z 
2022-12-01T10:41:27.6892520Z Running tests...
2022-12-01T10:41:27.6892785Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6893096Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6893347Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-12-01T10:41:27.6893367Z 
2022-12-01T10:41:27.6893626Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6893740Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6893760Z 
2022-12-01T10:41:27.6893849Z OK (skipped=1)
2022-12-01T10:41:27.6893868Z 
2022-12-01T10:41:27.6893992Z Generating XML reports...
2022-12-01T10:41:27.6894440Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102822.xml
2022-12-01T10:41:27.6894818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6894994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6895373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6895566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6895586Z 
2022-12-01T10:41:27.6895694Z Running tests...
2022-12-01T10:41:27.6895957Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6896248Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6896507Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI with timeout (0.002s)
2022-12-01T10:41:27.6896527Z 
2022-12-01T10:41:27.6896791Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6896955Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6896975Z 
2022-12-01T10:41:27.6897086Z OK (skipped=1)
2022-12-01T10:41:27.6897105Z 
2022-12-01T10:41:27.6897227Z Generating XML reports...
2022-12-01T10:41:27.6897677Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102824.xml
2022-12-01T10:41:27.6898049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6898205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6898585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6898778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6898798Z 
2022-12-01T10:41:27.6898907Z Running tests...
2022-12-01T10:41:27.6899172Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6899492Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6899761Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-12-01T10:41:27.6899781Z 
2022-12-01T10:41:27.6900042Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6900156Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6900176Z 
2022-12-01T10:41:27.6900264Z OK (skipped=1)
2022-12-01T10:41:27.6900282Z 
2022-12-01T10:41:27.6900407Z Generating XML reports...
2022-12-01T10:41:27.6900851Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102826.xml
2022-12-01T10:41:27.6901221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6901396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6901817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6902015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6902035Z 
2022-12-01T10:41:27.6902144Z Running tests...
2022-12-01T10:41:27.6902410Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6902703Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6902972Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6903195Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20723
2022-12-01T10:41:27.6903414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20724
2022-12-01T10:41:27.6903783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6903964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6904348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6904540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6904887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6905062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6905436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6905626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6905874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6906281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6906582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6906981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6907211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6907421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6907581Z skip: Skipped due to small world size. (3.820s)
2022-12-01T10:41:27.6907601Z 
2022-12-01T10:41:27.6907873Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6907986Z Ran 1 test in 3.820s
2022-12-01T10:41:27.6908006Z 
2022-12-01T10:41:27.6908112Z OK (skipped=1)
2022-12-01T10:41:27.6908131Z 
2022-12-01T10:41:27.6908254Z Generating XML reports...
2022-12-01T10:41:27.6908709Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102828.xml
2022-12-01T10:41:27.6909082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6909259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6909799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6909995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6910015Z 
2022-12-01T10:41:27.6910124Z Running tests...
2022-12-01T10:41:27.6910392Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6910704Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6910965Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-12-01T10:41:27.6910989Z 
2022-12-01T10:41:27.6911320Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6911442Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6911462Z 
2022-12-01T10:41:27.6911552Z OK (skipped=1)
2022-12-01T10:41:27.6911590Z 
2022-12-01T10:41:27.6911696Z Generating XML reports...
2022-12-01T10:41:27.6912147Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102834.xml
2022-12-01T10:41:27.6912519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6912695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6913076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6913267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6913290Z 
2022-12-01T10:41:27.6913401Z Running tests...
2022-12-01T10:41:27.6913671Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6913964Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6914222Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6914443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21089
2022-12-01T10:41:27.6914661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21090
2022-12-01T10:41:27.6915030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6915205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6915586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6915847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6916200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6916375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6916747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6916937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6917183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6917428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6917831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6918239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6918471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6918683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6918842Z skip: Skipped due to small world size. (4.086s)
2022-12-01T10:41:27.6918863Z 
2022-12-01T10:41:27.6919130Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6919243Z Ran 1 test in 4.086s
2022-12-01T10:41:27.6919263Z 
2022-12-01T10:41:27.6919371Z OK (skipped=1)
2022-12-01T10:41:27.6919390Z 
2022-12-01T10:41:27.6919513Z Generating XML reports...
2022-12-01T10:41:27.6919963Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102836.xml
2022-12-01T10:41:27.6920338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6920568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6920944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6921137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6921157Z 
2022-12-01T10:41:27.6921263Z Running tests...
2022-12-01T10:41:27.6921529Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6921839Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6922118Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T10:41:27.6922138Z 
2022-12-01T10:41:27.6922399Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6922516Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6922536Z 
2022-12-01T10:41:27.6922646Z OK (skipped=1)
2022-12-01T10:41:27.6922665Z 
2022-12-01T10:41:27.6922771Z Generating XML reports...
2022-12-01T10:41:27.6923219Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102842.xml
2022-12-01T10:41:27.6923591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6923767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6924146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6924336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6924356Z 
2022-12-01T10:41:27.6924464Z Running tests...
2022-12-01T10:41:27.6924727Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6925078Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6925354Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T10:41:27.6925375Z 
2022-12-01T10:41:27.6925638Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6925748Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6925768Z 
2022-12-01T10:41:27.6925873Z OK (skipped=1)
2022-12-01T10:41:27.6925892Z 
2022-12-01T10:41:27.6926015Z Generating XML reports...
2022-12-01T10:41:27.6926461Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102845.xml
2022-12-01T10:41:27.6926831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6927009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6927377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6927573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6927595Z 
2022-12-01T10:41:27.6927707Z Running tests...
2022-12-01T10:41:27.6927977Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6928289Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6928564Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T10:41:27.6928584Z 
2022-12-01T10:41:27.6928847Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6928960Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6928980Z 
2022-12-01T10:41:27.6929084Z OK (skipped=1)
2022-12-01T10:41:27.6929103Z 
2022-12-01T10:41:27.6929208Z Generating XML reports...
2022-12-01T10:41:27.6929700Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102847.xml
2022-12-01T10:41:27.6930085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6930261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6930638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6930829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6930849Z 
2022-12-01T10:41:27.6930957Z Running tests...
2022-12-01T10:41:27.6931221Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6931513Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6931775Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T10:41:27.6931799Z 
2022-12-01T10:41:27.6932066Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6932181Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6932202Z 
2022-12-01T10:41:27.6932308Z OK (skipped=1)
2022-12-01T10:41:27.6932326Z 
2022-12-01T10:41:27.6932448Z Generating XML reports...
2022-12-01T10:41:27.6932894Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102849.xml
2022-12-01T10:41:27.6933270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6933445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6933804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6933997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6934061Z 
2022-12-01T10:41:27.6934173Z Running tests...
2022-12-01T10:41:27.6934438Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6934750Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6935015Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T10:41:27.6935035Z 
2022-12-01T10:41:27.6935288Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6935401Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6935421Z 
2022-12-01T10:41:27.6935528Z OK (skipped=1)
2022-12-01T10:41:27.6935547Z 
2022-12-01T10:41:27.6935650Z Generating XML reports...
2022-12-01T10:41:27.6936187Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102851.xml
2022-12-01T10:41:27.6936594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6936825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6937222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6937610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6937631Z 
2022-12-01T10:41:27.6937783Z Running tests...
2022-12-01T10:41:27.6938085Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6938434Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6938691Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6938763Z 
2022-12-01T10:41:27.6939007Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6939162Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6939187Z 
2022-12-01T10:41:27.6939328Z OK (skipped=1)
2022-12-01T10:41:27.6939347Z 
2022-12-01T10:41:27.6939553Z Generating XML reports...
2022-12-01T10:41:27.6940092Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102853.xml
2022-12-01T10:41:27.6940502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6940717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6941154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6941329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6941411Z 
2022-12-01T10:41:27.6941504Z Running tests...
2022-12-01T10:41:27.6941803Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6942148Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6942452Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-12-01T10:41:27.6942474Z 
2022-12-01T10:41:27.6942811Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6942959Z Ran 1 test in 0.003s
2022-12-01T10:41:27.6942979Z 
2022-12-01T10:41:27.6943119Z OK (skipped=1)
2022-12-01T10:41:27.6943139Z 
2022-12-01T10:41:27.6943303Z Generating XML reports...
2022-12-01T10:41:27.6943732Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102855.xml
2022-12-01T10:41:27.6944139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6944348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6944763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6945047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6945067Z 
2022-12-01T10:41:27.6945244Z Running tests...
2022-12-01T10:41:27.6945560Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6945941Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6946250Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6946271Z 
2022-12-01T10:41:27.6946516Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6946665Z Ran 1 test in 0.003s
2022-12-01T10:41:27.6946687Z 
2022-12-01T10:41:27.6946828Z OK (skipped=1)
2022-12-01T10:41:27.6946848Z 
2022-12-01T10:41:27.6947005Z Generating XML reports...
2022-12-01T10:41:27.6947491Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102857.xml
2022-12-01T10:41:27.6947950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6948167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6948585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6948761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6948834Z 
2022-12-01T10:41:27.6948925Z Running tests...
2022-12-01T10:41:27.6949220Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6949814Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6950117Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6950143Z 
2022-12-01T10:41:27.6950534Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6950744Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6950764Z 
2022-12-01T10:41:27.6950942Z OK (skipped=1)
2022-12-01T10:41:27.6950962Z 
2022-12-01T10:41:27.6951121Z Generating XML reports...
2022-12-01T10:41:27.6951555Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102859.xml
2022-12-01T10:41:27.6951976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6952185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6952606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6952842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6952863Z 
2022-12-01T10:41:27.6953010Z Running tests...
2022-12-01T10:41:27.6953352Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6953702Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6954004Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6954024Z 
2022-12-01T10:41:27.6954267Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6954414Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6954434Z 
2022-12-01T10:41:27.6954574Z OK (skipped=1)
2022-12-01T10:41:27.6954594Z 
2022-12-01T10:41:27.6954760Z Generating XML reports...
2022-12-01T10:41:27.6955236Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102901.xml
2022-12-01T10:41:27.6955645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6956005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6956432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6956663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6956685Z 
2022-12-01T10:41:27.6956780Z Running tests...
2022-12-01T10:41:27.6957090Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6957436Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6957747Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6957768Z 
2022-12-01T10:41:27.6958060Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6958206Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6958225Z 
2022-12-01T10:41:27.6958405Z OK (skipped=1)
2022-12-01T10:41:27.6958425Z 
2022-12-01T10:41:27.6958588Z Generating XML reports...
2022-12-01T10:41:27.6959023Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102903.xml
2022-12-01T10:41:27.6959446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6959658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6960073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6960300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6960320Z 
2022-12-01T10:41:27.6960463Z Running tests...
2022-12-01T10:41:27.6960761Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6961171Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6961533Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6961556Z 
2022-12-01T10:41:27.6961813Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6961963Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6961983Z 
2022-12-01T10:41:27.6962122Z OK (skipped=1)
2022-12-01T10:41:27.6962142Z 
2022-12-01T10:41:27.6962296Z Generating XML reports...
2022-12-01T10:41:27.6962779Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102905.xml
2022-12-01T10:41:27.6963186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6963394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6963853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6964093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6964114Z 
2022-12-01T10:41:27.6964206Z Running tests...
2022-12-01T10:41:27.6964502Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6964850Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6965153Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:41:27.6965174Z 
2022-12-01T10:41:27.6965468Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6965614Z Ran 1 test in 0.002s
2022-12-01T10:41:27.6965635Z 
2022-12-01T10:41:27.6965821Z OK (skipped=1)
2022-12-01T10:41:27.6965840Z 
2022-12-01T10:41:27.6966036Z Generating XML reports...
2022-12-01T10:41:27.6966466Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102907.xml
2022-12-01T10:41:27.6966937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6967152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6967570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6967796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6967816Z 
2022-12-01T10:41:27.6967968Z Running tests...
2022-12-01T10:41:27.6968268Z ----------------------------------------------------------------------
2022-12-01T10:41:27.6968610Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.6968924Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.6969132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22588
2022-12-01T10:41:27.6969395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22589
2022-12-01T10:41:27.6969804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6970016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6970437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6970662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6971079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.6971293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.6971731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.6971912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.6972236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.6972522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.6972973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6973411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.6973676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.6974046Z STAGE:2022-12-01 10:29:13 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6974307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.6974630Z STAGE:2022-12-01 10:29:13 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6974943Z [1669890555.330107] [7ff5451e42d4:22589:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6975219Z [1669890555.330107] [7ff5451e42d4:22589:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6975492Z [1669890555.321279] [7ff5451e42d4:22588:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.6975767Z [1669890555.321279] [7ff5451e42d4:22588:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.6976358Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6976478Z 
2022-12-01T10:41:27.6976861Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6977239Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6977599Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6978001Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6978374Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6978689Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6979055Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6979418Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6979786Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6980142Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6980509Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6980946Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6981375Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6981685Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6982050Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6982418Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6982877Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6983250Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6983619Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6983992Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6984353Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6984746Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6985063Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6985428Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6985790Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6986154Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6986521Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6986897Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6987256Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6987615Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6988010Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6988326Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6988751Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6989142Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.6989768Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.6990155Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7021142Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7021577Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7021913Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7022232Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7022562Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7022872Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7023188Z STAGE:2022-12-01 10:29:15 22588:22588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7023495Z STAGE:2022-12-01 10:29:15 22589:22589 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7023589Z ok (7.123s)
2022-12-01T10:41:27.7023611Z 
2022-12-01T10:41:27.7023870Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7023974Z Ran 1 test in 7.123s
2022-12-01T10:41:27.7023995Z 
2022-12-01T10:41:27.7024076Z OK
2022-12-01T10:41:27.7024095Z 
2022-12-01T10:41:27.7024210Z Generating XML reports...
2022-12-01T10:41:27.7024653Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102909.xml
2022-12-01T10:41:27.7025176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7025359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7025735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7025918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7025939Z 
2022-12-01T10:41:27.7026035Z Running tests...
2022-12-01T10:41:27.7026290Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7026589Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7026865Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and Nccl backend supports CUDA allReduce (0.003s)
2022-12-01T10:41:27.7026891Z 
2022-12-01T10:41:27.7027146Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7027247Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7027268Z 
2022-12-01T10:41:27.7027358Z OK (skipped=1)
2022-12-01T10:41:27.7027384Z 
2022-12-01T10:41:27.7027490Z Generating XML reports...
2022-12-01T10:41:27.7027928Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102919.xml
2022-12-01T10:41:27.7028293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7028458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7028826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7029007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7029096Z 
2022-12-01T10:41:27.7029201Z Running tests...
2022-12-01T10:41:27.7029694Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7030006Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7030264Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7030476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23109
2022-12-01T10:41:27.7030683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23110
2022-12-01T10:41:27.7031045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7031210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7031579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7031763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7032115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7032282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7032649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7032828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7033063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7033296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7033684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7034071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7034372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7034607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7034820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7035043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7035432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7035809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7036341Z STAGE:2022-12-01 10:29:24 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:29:24 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7036367Z 
2022-12-01T10:41:27.7036591Z [1669890566.857755] [7ff5451e42d4:23110:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7036821Z [1669890566.857755] [7ff5451e42d4:23110:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7037033Z [1669890566.822104] [7ff5451e42d4:23109:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7037251Z [1669890566.822104] [7ff5451e42d4:23109:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7037790Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7037870Z 
2022-12-01T10:41:27.7038200Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7038527Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7038903Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7039208Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7039523Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7039843Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7040172Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7040499Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7040827Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7041128Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7041447Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7041770Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7042090Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7042409Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7042733Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7043057Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7043423Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7043749Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7044053Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7044377Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7044691Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7045007Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7045330Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7045651Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7045977Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7046295Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7046621Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7046924Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7047242Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7047558Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7047880Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7048203Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7048580Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7048894Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7049219Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7049540Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7049837Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7050153Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7050477Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7050799Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7051124Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7051440Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7051764Z STAGE:2022-12-01 10:29:27 23109:23109 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7052084Z STAGE:2022-12-01 10:29:27 23110:23110 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7052189Z ok (7.106s)
2022-12-01T10:41:27.7052209Z 
2022-12-01T10:41:27.7052458Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7052576Z Ran 1 test in 7.107s
2022-12-01T10:41:27.7052596Z 
2022-12-01T10:41:27.7052687Z OK
2022-12-01T10:41:27.7052706Z 
2022-12-01T10:41:27.7052831Z Generating XML reports...
2022-12-01T10:41:27.7053285Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102921.xml
2022-12-01T10:41:27.7053710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7053892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7054276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7054453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7054492Z 
2022-12-01T10:41:27.7054582Z Running tests...
2022-12-01T10:41:27.7054851Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7055164Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7055420Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7055645Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23495
2022-12-01T10:41:27.7055867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23496
2022-12-01T10:41:27.7056239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7056419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7056782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7056973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7057343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7057519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7057899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7058148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7058394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7058637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7059023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7059423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7059654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7059880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7060040Z skip: Skipped due to small world size. (4.073s)
2022-12-01T10:41:27.7060064Z 
2022-12-01T10:41:27.7060333Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7060448Z Ran 1 test in 4.073s
2022-12-01T10:41:27.7060468Z 
2022-12-01T10:41:27.7060577Z OK (skipped=1)
2022-12-01T10:41:27.7060596Z 
2022-12-01T10:41:27.7060725Z Generating XML reports...
2022-12-01T10:41:27.7061158Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102930.xml
2022-12-01T10:41:27.7061531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7061708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7062090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7062281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7062305Z 
2022-12-01T10:41:27.7062414Z Running tests...
2022-12-01T10:41:27.7062721Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7063044Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7063309Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7063511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23770
2022-12-01T10:41:27.7063731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23771
2022-12-01T10:41:27.7064099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7064277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7064651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7064844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7065217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7065396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7065756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7065947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7066194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7066438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7066842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7067243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7067525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7067755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7068546Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:41:27.7068663Z   warnings.warn(
2022-12-01T10:41:27.7069667Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:41:27.7069796Z   warnings.warn(
2022-12-01T10:41:27.7070037Z [1669890581.933754] [7ff5451e42d4:23770:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7070279Z [1669890581.933754] [7ff5451e42d4:23770:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7070505Z [1669890581.938113] [7ff5451e42d4:23771:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7070745Z [1669890581.938113] [7ff5451e42d4:23771:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7070849Z ok (5.962s)
2022-12-01T10:41:27.7070869Z 
2022-12-01T10:41:27.7071145Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7071260Z Ran 1 test in 5.962s
2022-12-01T10:41:27.7071280Z 
2022-12-01T10:41:27.7071354Z OK
2022-12-01T10:41:27.7071378Z 
2022-12-01T10:41:27.7071505Z Generating XML reports...
2022-12-01T10:41:27.7072033Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102936.xml
2022-12-01T10:41:27.7072418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7072594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7072975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7073168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7073190Z 
2022-12-01T10:41:27.7073301Z Running tests...
2022-12-01T10:41:27.7073572Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7073865Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7074280Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7074301Z 
2022-12-01T10:41:27.7074565Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7074679Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7074700Z 
2022-12-01T10:41:27.7074808Z OK (skipped=1)
2022-12-01T10:41:27.7074828Z 
2022-12-01T10:41:27.7074952Z Generating XML reports...
2022-12-01T10:41:27.7075402Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102945.xml
2022-12-01T10:41:27.7075774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7075952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7076320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7076598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7076622Z 
2022-12-01T10:41:27.7076731Z Running tests...
2022-12-01T10:41:27.7076999Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7077312Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7077790Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7077811Z 
2022-12-01T10:41:27.7078071Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7079430Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7079451Z 
2022-12-01T10:41:27.7079560Z OK (skipped=1)
2022-12-01T10:41:27.7079580Z 
2022-12-01T10:41:27.7079687Z Generating XML reports...
2022-12-01T10:41:27.7080141Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102947.xml
2022-12-01T10:41:27.7080520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7080747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7081131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7081323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7081343Z 
2022-12-01T10:41:27.7081451Z Running tests...
2022-12-01T10:41:27.7081714Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7082005Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7082489Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7082514Z 
2022-12-01T10:41:27.7082835Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7082957Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7082977Z 
2022-12-01T10:41:27.7083084Z OK (skipped=1)
2022-12-01T10:41:27.7083105Z 
2022-12-01T10:41:27.7083228Z Generating XML reports...
2022-12-01T10:41:27.7083672Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102949.xml
2022-12-01T10:41:27.7084043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7084217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7084594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7084768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7084792Z 
2022-12-01T10:41:27.7084901Z Running tests...
2022-12-01T10:41:27.7085167Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7085475Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7085740Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7085960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24449
2022-12-01T10:41:27.7086179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24450
2022-12-01T10:41:27.7086552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7086710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7087092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7087339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7087708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7087882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7088255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7088446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7088692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7088938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7089322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7089731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7089962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7090193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7090452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkm1bd7n
2022-12-01T10:41:27.7090725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkm1bd7n/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7090978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qu5pzdf
2022-12-01T10:41:27.7091248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qu5pzdf/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7091483Z [1669890596.316009] [7ff5451e42d4:24450:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7091755Z [1669890596.316009] [7ff5451e42d4:24450:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7091992Z [1669890596.313678] [7ff5451e42d4:24449:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7092222Z [1669890596.313678] [7ff5451e42d4:24449:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7092460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7092695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7092798Z ok (6.377s)
2022-12-01T10:41:27.7092818Z 
2022-12-01T10:41:27.7093092Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7093205Z Ran 1 test in 6.378s
2022-12-01T10:41:27.7093225Z 
2022-12-01T10:41:27.7093299Z OK
2022-12-01T10:41:27.7093341Z 
2022-12-01T10:41:27.7093447Z Generating XML reports...
2022-12-01T10:41:27.7093902Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102951.xml
2022-12-01T10:41:27.7094277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7094454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7094834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7095025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7095045Z 
2022-12-01T10:41:27.7095158Z Running tests...
2022-12-01T10:41:27.7095422Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7095717Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7096052Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7096273Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24854
2022-12-01T10:41:27.7096491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24855
2022-12-01T10:41:27.7096866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7097043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7097421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7097611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7097975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7098134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7098521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7098711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7098956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7099201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7099599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7099997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7100226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7100435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7100744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv3o2mlxz
2022-12-01T10:41:27.7101024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv3o2mlxz/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7101278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl4g7zgom
2022-12-01T10:41:27.7101544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl4g7zgom/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7101778Z [1669890604.952665] [7ff5451e42d4:24854:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7102020Z [1669890604.952665] [7ff5451e42d4:24854:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7102251Z [1669890604.957897] [7ff5451e42d4:24855:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7102491Z [1669890604.957897] [7ff5451e42d4:24855:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7102729Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7102950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7103182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7103415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7103518Z ok (6.603s)
2022-12-01T10:41:27.7103539Z 
2022-12-01T10:41:27.7103810Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7103923Z Ran 1 test in 6.603s
2022-12-01T10:41:27.7103943Z 
2022-12-01T10:41:27.7104038Z OK
2022-12-01T10:41:27.7104057Z 
2022-12-01T10:41:27.7104179Z Generating XML reports...
2022-12-01T10:41:27.7104671Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102959.xml
2022-12-01T10:41:27.7105043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7105221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7105596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7105787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7105807Z 
2022-12-01T10:41:27.7105914Z Running tests...
2022-12-01T10:41:27.7106178Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7106486Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7106740Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7107498Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.729s)
2022-12-01T10:41:27.7107538Z 
2022-12-01T10:41:27.7107782Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7107894Z Ran 1 test in 1.729s
2022-12-01T10:41:27.7107914Z 
2022-12-01T10:41:27.7108022Z OK (skipped=1)
2022-12-01T10:41:27.7108043Z 
2022-12-01T10:41:27.7108165Z Generating XML reports...
2022-12-01T10:41:27.7108609Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103008.xml
2022-12-01T10:41:27.7108982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7109162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7109869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7110060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7110099Z 
2022-12-01T10:41:27.7110190Z Running tests...
2022-12-01T10:41:27.7110464Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7110774Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7111063Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7111811Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.713s)
2022-12-01T10:41:27.7111836Z 
2022-12-01T10:41:27.7112096Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7112209Z Ran 1 test in 1.713s
2022-12-01T10:41:27.7112229Z 
2022-12-01T10:41:27.7112335Z OK (skipped=1)
2022-12-01T10:41:27.7112353Z 
2022-12-01T10:41:27.7112476Z Generating XML reports...
2022-12-01T10:41:27.7112904Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103012.xml
2022-12-01T10:41:27.7113272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7113447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7113821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7114084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7114105Z 
2022-12-01T10:41:27.7114219Z Running tests...
2022-12-01T10:41:27.7114490Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7114804Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7115221Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-12-01T10:41:27.7115261Z 
2022-12-01T10:41:27.7115507Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7115619Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7115639Z 
2022-12-01T10:41:27.7115749Z OK (skipped=1)
2022-12-01T10:41:27.7115768Z 
2022-12-01T10:41:27.7115893Z Generating XML reports...
2022-12-01T10:41:27.7116340Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103016.xml
2022-12-01T10:41:27.7116719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7116893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7117277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7117453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7117494Z 
2022-12-01T10:41:27.7117586Z Running tests...
2022-12-01T10:41:27.7117851Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7118161Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7118468Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7118691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25659
2022-12-01T10:41:27.7118958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25660
2022-12-01T10:41:27.7119343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7119519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7119878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7120069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7120434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7120610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7120987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7121185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7121434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7121682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7122083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7122464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7122695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7122923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7123178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiocit3q5
2022-12-01T10:41:27.7123503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiocit3q5/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7123759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0ypbuh_
2022-12-01T10:41:27.7124028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0ypbuh_/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7124264Z [1669890623.959160] [7ff5451e42d4:25659:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7124505Z [1669890623.959160] [7ff5451e42d4:25659:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7124712Z [1669890623.961447] [7ff5451e42d4:25660:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7124943Z [1669890623.961447] [7ff5451e42d4:25660:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7125051Z ok (6.006s)
2022-12-01T10:41:27.7125072Z 
2022-12-01T10:41:27.7125348Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7125462Z Ran 1 test in 6.006s
2022-12-01T10:41:27.7125482Z 
2022-12-01T10:41:27.7125575Z OK
2022-12-01T10:41:27.7125594Z 
2022-12-01T10:41:27.7125718Z Generating XML reports...
2022-12-01T10:41:27.7126164Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103018.xml
2022-12-01T10:41:27.7126520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7126698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7127079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7127270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7127294Z 
2022-12-01T10:41:27.7127402Z Running tests...
2022-12-01T10:41:27.7127713Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7128036Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7128298Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7128518Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26030
2022-12-01T10:41:27.7128716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26031
2022-12-01T10:41:27.7129089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7129263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7129639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7129835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7130202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7130378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7130759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7130933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7131180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7131425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7131825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7132284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7132515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7132742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7132997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_a3j36yj
2022-12-01T10:41:27.7133265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_a3j36yj/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7133504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnn9ujpg
2022-12-01T10:41:27.7133773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnn9ujpg/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7134008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7134245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7134483Z [1669890632.150666] [7ff5451e42d4:26030:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7134724Z [1669890632.150666] [7ff5451e42d4:26030:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7134947Z [1669890632.158022] [7ff5451e42d4:26031:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7135176Z [1669890632.158022] [7ff5451e42d4:26031:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7135278Z ok (6.570s)
2022-12-01T10:41:27.7135299Z 
2022-12-01T10:41:27.7135552Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7135666Z Ran 1 test in 6.570s
2022-12-01T10:41:27.7135685Z 
2022-12-01T10:41:27.7135781Z OK
2022-12-01T10:41:27.7135801Z 
2022-12-01T10:41:27.7135924Z Generating XML reports...
2022-12-01T10:41:27.7136414Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103027.xml
2022-12-01T10:41:27.7136799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7136977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7137356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7137548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7137569Z 
2022-12-01T10:41:27.7137658Z Running tests...
2022-12-01T10:41:27.7137922Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7138230Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7138677Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.004s)
2022-12-01T10:41:27.7138698Z 
2022-12-01T10:41:27.7138958Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7139073Z Ran 1 test in 0.004s
2022-12-01T10:41:27.7139094Z 
2022-12-01T10:41:27.7139202Z OK (skipped=1)
2022-12-01T10:41:27.7139221Z 
2022-12-01T10:41:27.7139344Z Generating XML reports...
2022-12-01T10:41:27.7139790Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103035.xml
2022-12-01T10:41:27.7140146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7140321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7140703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7140963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7140984Z 
2022-12-01T10:41:27.7141093Z Running tests...
2022-12-01T10:41:27.7141356Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7141665Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7142093Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.004s)
2022-12-01T10:41:27.7142114Z 
2022-12-01T10:41:27.7142374Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7142469Z Ran 1 test in 0.004s
2022-12-01T10:41:27.7142489Z 
2022-12-01T10:41:27.7142596Z OK (skipped=1)
2022-12-01T10:41:27.7142615Z 
2022-12-01T10:41:27.7142738Z Generating XML reports...
2022-12-01T10:41:27.7143178Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103037.xml
2022-12-01T10:41:27.7143554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7143730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7144109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7144299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7144319Z 
2022-12-01T10:41:27.7144426Z Running tests...
2022-12-01T10:41:27.7144671Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7144984Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7145243Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7145466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26628
2022-12-01T10:41:27.7145753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26629
2022-12-01T10:41:27.7146138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7146319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7146696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7146867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7147235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7147412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7147789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7147986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7148232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7148476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7148876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7149271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7149805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7150074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfy8wwc39
2022-12-01T10:41:27.7150345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfy8wwc39/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7150664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7150918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37oeq8wb
2022-12-01T10:41:27.7151184Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37oeq8wb/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7151418Z [1669890645.231193] [7ff5451e42d4:26628:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7151659Z [1669890645.231193] [7ff5451e42d4:26628:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7152573Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7152814Z [1669890645.248740] [7ff5451e42d4:26629:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7153037Z [1669890645.248740] [7ff5451e42d4:26629:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7153918Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7155140Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:41:27.7155388Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T10:41:27.7156546Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:41:27.7156788Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T10:41:27.7157026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7157261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7158150Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7159025Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7159943Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7160808Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7161667Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7162525Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7163377Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7164274Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7165140Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7165991Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:41:27.7166100Z ok (5.947s)
2022-12-01T10:41:27.7166121Z 
2022-12-01T10:41:27.7166391Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7166505Z Ran 1 test in 5.948s
2022-12-01T10:41:27.7166526Z 
2022-12-01T10:41:27.7166618Z OK
2022-12-01T10:41:27.7166637Z 
2022-12-01T10:41:27.7166761Z Generating XML reports...
2022-12-01T10:41:27.7167212Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103040.xml
2022-12-01T10:41:27.7167592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7167754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7168195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7168388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7168408Z 
2022-12-01T10:41:27.7168518Z Running tests...
2022-12-01T10:41:27.7168784Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7169096Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7169479Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.005s)
2022-12-01T10:41:27.7169500Z 
2022-12-01T10:41:27.7169761Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7169855Z Ran 1 test in 0.005s
2022-12-01T10:41:27.7169893Z 
2022-12-01T10:41:27.7169983Z OK (skipped=1)
2022-12-01T10:41:27.7170002Z 
2022-12-01T10:41:27.7170125Z Generating XML reports...
2022-12-01T10:41:27.7170577Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103048.xml
2022-12-01T10:41:27.7170946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7171124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7171502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7171693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7171713Z 
2022-12-01T10:41:27.7171822Z Running tests...
2022-12-01T10:41:27.7172065Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7172374Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7172786Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T10:41:27.7172861Z 
2022-12-01T10:41:27.7173143Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7173255Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7173275Z 
2022-12-01T10:41:27.7173383Z OK (skipped=1)
2022-12-01T10:41:27.7173402Z 
2022-12-01T10:41:27.7173526Z Generating XML reports...
2022-12-01T10:41:27.7173967Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103050.xml
2022-12-01T10:41:27.7174331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7174490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7174865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7175057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7175082Z 
2022-12-01T10:41:27.7175191Z Running tests...
2022-12-01T10:41:27.7175457Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7175769Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7176042Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7176787Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.611s)
2022-12-01T10:41:27.7176809Z 
2022-12-01T10:41:27.7177072Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7177185Z Ran 1 test in 1.611s
2022-12-01T10:41:27.7177252Z 
2022-12-01T10:41:27.7177348Z OK (skipped=1)
2022-12-01T10:41:27.7177367Z 
2022-12-01T10:41:27.7177495Z Generating XML reports...
2022-12-01T10:41:27.7177947Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103052.xml
2022-12-01T10:41:27.7178320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7178495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7178872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7179062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7179083Z 
2022-12-01T10:41:27.7179193Z Running tests...
2022-12-01T10:41:27.7179437Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7179749Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7180029Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7180815Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.705s)
2022-12-01T10:41:27.7180837Z 
2022-12-01T10:41:27.7181101Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7181214Z Ran 1 test in 1.705s
2022-12-01T10:41:27.7181233Z 
2022-12-01T10:41:27.7181341Z OK (skipped=1)
2022-12-01T10:41:27.7181360Z 
2022-12-01T10:41:27.7181485Z Generating XML reports...
2022-12-01T10:41:27.7181930Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103056.xml
2022-12-01T10:41:27.7182349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7182513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7182896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7183086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7183107Z 
2022-12-01T10:41:27.7183215Z Running tests...
2022-12-01T10:41:27.7183478Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7183789Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7184078Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7184296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27506
2022-12-01T10:41:27.7184505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27507
2022-12-01T10:41:27.7184884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7185060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7185440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7185632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7185998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7186173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7186550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7186796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7187029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7187280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7187687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7188084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7188312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7188539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7188779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7189024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7189650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7190043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7190305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpop670b8v
2022-12-01T10:41:27.7190576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpop670b8v/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7190830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp38my9lz
2022-12-01T10:41:27.7191096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp38my9lz/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7191331Z [1669890665.340473] [7ff5451e42d4:27506:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7191667Z [1669890665.340473] [7ff5451e42d4:27506:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7191905Z [1669890665.344683] [7ff5451e42d4:27507:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7192136Z [1669890665.344683] [7ff5451e42d4:27507:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7192357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7192594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7192825Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7193057Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7193163Z ok (6.700s)
2022-12-01T10:41:27.7193184Z 
2022-12-01T10:41:27.7193465Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7193578Z Ran 1 test in 6.700s
2022-12-01T10:41:27.7193598Z 
2022-12-01T10:41:27.7193691Z OK
2022-12-01T10:41:27.7193710Z 
2022-12-01T10:41:27.7193834Z Generating XML reports...
2022-12-01T10:41:27.7194261Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103100.xml
2022-12-01T10:41:27.7194637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7194813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7195193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7195384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7195469Z 
2022-12-01T10:41:27.7195583Z Running tests...
2022-12-01T10:41:27.7195857Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7196172Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7196431Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7196653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27833
2022-12-01T10:41:27.7196869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27834
2022-12-01T10:41:27.7197241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7197420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7197802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7197996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7198373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7198548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7198907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7199098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7199342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7199585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7199985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7200437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7200673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7200952Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:41:27.7201178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7201429Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:41:27.7201688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1gfkbnv
2022-12-01T10:41:27.7201959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1gfkbnv/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7202209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9uidgb2
2022-12-01T10:41:27.7202481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9uidgb2/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7202715Z [1669890674.286016] [7ff5451e42d4:27833:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7202957Z [1669890674.286016] [7ff5451e42d4:27833:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7203184Z [1669890674.283265] [7ff5451e42d4:27834:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7203417Z [1669890674.283265] [7ff5451e42d4:27834:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7203635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7203873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7204177Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7204414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7204691Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:41:27.7204968Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:41:27.7205243Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:41:27.7205516Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:41:27.7205747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7205962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7206202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7206434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7206708Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:41:27.7206979Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:41:27.7207253Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T10:41:27.7207520Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T10:41:27.7207751Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7207980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7208242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7208482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7208587Z ok (7.187s)
2022-12-01T10:41:27.7208608Z 
2022-12-01T10:41:27.7208880Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7208993Z Ran 1 test in 7.188s
2022-12-01T10:41:27.7209014Z 
2022-12-01T10:41:27.7209105Z OK
2022-12-01T10:41:27.7209124Z 
2022-12-01T10:41:27.7209247Z Generating XML reports...
2022-12-01T10:41:27.7209700Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103109.xml
2022-12-01T10:41:27.7210057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7210234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7210624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7210817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7210837Z 
2022-12-01T10:41:27.7210945Z Running tests...
2022-12-01T10:41:27.7211210Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7211519Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7211787Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7212535Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.693s)
2022-12-01T10:41:27.7212604Z 
2022-12-01T10:41:27.7212877Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7212972Z Ran 1 test in 1.693s
2022-12-01T10:41:27.7212992Z 
2022-12-01T10:41:27.7213103Z OK (skipped=1)
2022-12-01T10:41:27.7213123Z 
2022-12-01T10:41:27.7213246Z Generating XML reports...
2022-12-01T10:41:27.7213695Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103118.xml
2022-12-01T10:41:27.7214066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7214246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7214625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7214894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7214914Z 
2022-12-01T10:41:27.7215009Z Running tests...
2022-12-01T10:41:27.7215277Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7215590Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7216054Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI: No such file or directory: '/tmp/checkpoint.pt' (0.002s)
2022-12-01T10:41:27.7216074Z 
2022-12-01T10:41:27.7216328Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7216440Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7216460Z 
2022-12-01T10:41:27.7216566Z OK (skipped=1)
2022-12-01T10:41:27.7216585Z 
2022-12-01T10:41:27.7216708Z Generating XML reports...
2022-12-01T10:41:27.7217152Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103122.xml
2022-12-01T10:41:27.7217504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7217731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7218121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7218315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7218334Z 
2022-12-01T10:41:27.7218443Z Running tests...
2022-12-01T10:41:27.7218704Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7219017Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7219408Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7219429Z 
2022-12-01T10:41:27.7219694Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7219794Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7219832Z 
2022-12-01T10:41:27.7219924Z OK (skipped=1)
2022-12-01T10:41:27.7219943Z 
2022-12-01T10:41:27.7220067Z Generating XML reports...
2022-12-01T10:41:27.7220510Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103124.xml
2022-12-01T10:41:27.7220878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7221052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7221429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7221621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7221642Z 
2022-12-01T10:41:27.7221750Z Running tests...
2022-12-01T10:41:27.7221997Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7222369Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7222761Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7222781Z 
2022-12-01T10:41:27.7223039Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7223150Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7223169Z 
2022-12-01T10:41:27.7223277Z OK (skipped=1)
2022-12-01T10:41:27.7223296Z 
2022-12-01T10:41:27.7223418Z Generating XML reports...
2022-12-01T10:41:27.7223864Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103126.xml
2022-12-01T10:41:27.7224238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7224401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7224784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7224974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7224994Z 
2022-12-01T10:41:27.7225104Z Running tests...
2022-12-01T10:41:27.7225367Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7225678Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7226126Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7226147Z 
2022-12-01T10:41:27.7226406Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7226522Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7226542Z 
2022-12-01T10:41:27.7226674Z OK (skipped=1)
2022-12-01T10:41:27.7226715Z 
2022-12-01T10:41:27.7226828Z Generating XML reports...
2022-12-01T10:41:27.7227278Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103128.xml
2022-12-01T10:41:27.7227649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7227825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7228202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7228393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7228414Z 
2022-12-01T10:41:27.7228521Z Running tests...
2022-12-01T10:41:27.7228785Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7229088Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7229774Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7229798Z 
2022-12-01T10:41:27.7230072Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7230184Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7230204Z 
2022-12-01T10:41:27.7230312Z OK (skipped=1)
2022-12-01T10:41:27.7230331Z 
2022-12-01T10:41:27.7230454Z Generating XML reports...
2022-12-01T10:41:27.7230896Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103130.xml
2022-12-01T10:41:27.7231267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7231533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7231916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7232091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7232112Z 
2022-12-01T10:41:27.7232224Z Running tests...
2022-12-01T10:41:27.7232486Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7232793Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7233235Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7233257Z 
2022-12-01T10:41:27.7233521Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7233638Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7233658Z 
2022-12-01T10:41:27.7233767Z OK (skipped=1)
2022-12-01T10:41:27.7233787Z 
2022-12-01T10:41:27.7233910Z Generating XML reports...
2022-12-01T10:41:27.7234337Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103132.xml
2022-12-01T10:41:27.7234710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7234884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7235261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7235451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7235471Z 
2022-12-01T10:41:27.7235579Z Running tests...
2022-12-01T10:41:27.7235845Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7236211Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7236663Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7236684Z 
2022-12-01T10:41:27.7236931Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7237044Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7237064Z 
2022-12-01T10:41:27.7237170Z OK (skipped=1)
2022-12-01T10:41:27.7237190Z 
2022-12-01T10:41:27.7237316Z Generating XML reports...
2022-12-01T10:41:27.7237763Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103134.xml
2022-12-01T10:41:27.7238133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7238313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7238694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7238884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7238904Z 
2022-12-01T10:41:27.7238994Z Running tests...
2022-12-01T10:41:27.7239256Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7239568Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7240013Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7240083Z 
2022-12-01T10:41:27.7240355Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7240469Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7240489Z 
2022-12-01T10:41:27.7240596Z OK (skipped=1)
2022-12-01T10:41:27.7240615Z 
2022-12-01T10:41:27.7240738Z Generating XML reports...
2022-12-01T10:41:27.7241180Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103137.xml
2022-12-01T10:41:27.7241533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7241711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7242093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7242285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7242304Z 
2022-12-01T10:41:27.7242415Z Running tests...
2022-12-01T10:41:27.7242681Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7242992Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7243435Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7243456Z 
2022-12-01T10:41:27.7243718Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7243812Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7243851Z 
2022-12-01T10:41:27.7243939Z OK (skipped=1)
2022-12-01T10:41:27.7243958Z 
2022-12-01T10:41:27.7244081Z Generating XML reports...
2022-12-01T10:41:27.7244526Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103139.xml
2022-12-01T10:41:27.7244944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7245126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7245505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7245698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7245718Z 
2022-12-01T10:41:27.7245825Z Running tests...
2022-12-01T10:41:27.7246070Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7246380Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7246824Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7246850Z 
2022-12-01T10:41:27.7247118Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7247230Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7247250Z 
2022-12-01T10:41:27.7247360Z OK (skipped=1)
2022-12-01T10:41:27.7247380Z 
2022-12-01T10:41:27.7247504Z Generating XML reports...
2022-12-01T10:41:27.7247947Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103141.xml
2022-12-01T10:41:27.7248322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7248500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7248861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7249057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7249122Z 
2022-12-01T10:41:27.7249236Z Running tests...
2022-12-01T10:41:27.7249504Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7249815Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7250258Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7250279Z 
2022-12-01T10:41:27.7250543Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7250655Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7250674Z 
2022-12-01T10:41:27.7250782Z OK (skipped=1)
2022-12-01T10:41:27.7250801Z 
2022-12-01T10:41:27.7250907Z Generating XML reports...
2022-12-01T10:41:27.7251352Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103143.xml
2022-12-01T10:41:27.7251734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7251910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7252288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7252479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7252499Z 
2022-12-01T10:41:27.7252607Z Running tests...
2022-12-01T10:41:27.7252873Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7253168Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7253561Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7253587Z 
2022-12-01T10:41:27.7253892Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7254009Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7254029Z 
2022-12-01T10:41:27.7254137Z OK (skipped=1)
2022-12-01T10:41:27.7254156Z 
2022-12-01T10:41:27.7254280Z Generating XML reports...
2022-12-01T10:41:27.7254725Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103145.xml
2022-12-01T10:41:27.7255093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7255268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7255642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7255816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7255840Z 
2022-12-01T10:41:27.7255948Z Running tests...
2022-12-01T10:41:27.7256216Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7256526Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7256910Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:41:27.7256931Z 
2022-12-01T10:41:27.7257190Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7257302Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7257322Z 
2022-12-01T10:41:27.7257432Z OK (skipped=1)
2022-12-01T10:41:27.7257451Z 
2022-12-01T10:41:27.7257574Z Generating XML reports...
2022-12-01T10:41:27.7258003Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103147.xml
2022-12-01T10:41:27.7258435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7258611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7258989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7259180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7259200Z 
2022-12-01T10:41:27.7259309Z Running tests...
2022-12-01T10:41:27.7259571Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7259883Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7260272Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7260311Z 
2022-12-01T10:41:27.7260556Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7260673Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7260697Z 
2022-12-01T10:41:27.7260806Z OK (skipped=1)
2022-12-01T10:41:27.7260825Z 
2022-12-01T10:41:27.7260948Z Generating XML reports...
2022-12-01T10:41:27.7261391Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103149.xml
2022-12-01T10:41:27.7261762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7261937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7262318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7262492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7262532Z 
2022-12-01T10:41:27.7262623Z Running tests...
2022-12-01T10:41:27.7262885Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7263242Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7263508Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7263727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29653
2022-12-01T10:41:27.7263943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29654
2022-12-01T10:41:27.7264317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7264494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7264854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7265045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7265422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7265596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7265973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7266163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7266409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7266654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7267038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7267440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7267740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7267969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7268226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74e6xv8l
2022-12-01T10:41:27.7268495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74e6xv8l/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7268746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ssdb93u
2022-12-01T10:41:27.7269012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ssdb93u/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7269245Z [1669890716.612579] [7ff5451e42d4:29653:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7269634Z [1669890716.612579] [7ff5451e42d4:29653:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7269879Z [1669890716.629667] [7ff5451e42d4:29654:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7270108Z [1669890716.629667] [7ff5451e42d4:29654:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7270212Z ok (6.889s)
2022-12-01T10:41:27.7270233Z 
2022-12-01T10:41:27.7270515Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7270628Z Ran 1 test in 6.890s
2022-12-01T10:41:27.7270648Z 
2022-12-01T10:41:27.7270740Z OK
2022-12-01T10:41:27.7270760Z 
2022-12-01T10:41:27.7270883Z Generating XML reports...
2022-12-01T10:41:27.7271337Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103151.xml
2022-12-01T10:41:27.7271691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7271873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7272327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7272531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7272552Z 
2022-12-01T10:41:27.7272662Z Running tests...
2022-12-01T10:41:27.7272925Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7273236Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7273508Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7273729Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29950
2022-12-01T10:41:27.7273930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29951
2022-12-01T10:41:27.7274314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7274490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7274865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7275054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7275418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7275595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7275973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7276147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7276395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7276716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7277122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7277515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7277746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7277970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7278227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2ke73kr
2022-12-01T10:41:27.7278498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2ke73kr/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7278733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu_weyvt
2022-12-01T10:41:27.7279007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu_weyvt/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7279242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7279476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7279709Z [1669890726.040399] [7ff5451e42d4:29950:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7279948Z [1669890726.040399] [7ff5451e42d4:29950:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7280172Z [1669890726.043553] [7ff5451e42d4:29951:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7280402Z [1669890726.043553] [7ff5451e42d4:29951:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7280903Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpiu_weyvt'>
2022-12-01T10:41:27.7281060Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:41:27.7281464Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpi2ke73kr'>
2022-12-01T10:41:27.7281626Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:41:27.7281729Z ok (6.486s)
2022-12-01T10:41:27.7281750Z 
2022-12-01T10:41:27.7282017Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7282129Z Ran 1 test in 6.486s
2022-12-01T10:41:27.7282149Z 
2022-12-01T10:41:27.7282242Z OK
2022-12-01T10:41:27.7282261Z 
2022-12-01T10:41:27.7282386Z Generating XML reports...
2022-12-01T10:41:27.7282815Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103200.xml
2022-12-01T10:41:27.7283193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7283371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7283754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7283946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7283966Z 
2022-12-01T10:41:27.7284073Z Running tests...
2022-12-01T10:41:27.7284342Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7284652Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7284915Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7285116Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30238
2022-12-01T10:41:27.7285415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30239
2022-12-01T10:41:27.7285790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7285967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7286342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7286533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7286897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7287070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7287431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7287627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7287875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7288122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7288524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7288921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7289150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7289403Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mxdque0
2022-12-01T10:41:27.7289675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mxdque0/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7289934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7290202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazfieujw
2022-12-01T10:41:27.7290472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazfieujw/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7290707Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7290944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7291177Z [1669890734.490967] [7ff5451e42d4:30238:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7291419Z [1669890734.490967] [7ff5451e42d4:30238:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7291644Z [1669890734.530916] [7ff5451e42d4:30239:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7291880Z [1669890734.530916] [7ff5451e42d4:30239:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7291966Z ok (6.419s)
2022-12-01T10:41:27.7292005Z 
2022-12-01T10:41:27.7292262Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7292377Z Ran 1 test in 6.419s
2022-12-01T10:41:27.7292396Z 
2022-12-01T10:41:27.7292488Z OK
2022-12-01T10:41:27.7292508Z 
2022-12-01T10:41:27.7292633Z Generating XML reports...
2022-12-01T10:41:27.7293078Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103209.xml
2022-12-01T10:41:27.7293454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7293631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7294016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7294241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7294262Z 
2022-12-01T10:41:27.7294370Z Running tests...
2022-12-01T10:41:27.7294636Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7294948Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7295215Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7295437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30644
2022-12-01T10:41:27.7295654Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30645
2022-12-01T10:41:27.7296025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7296189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7296577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7296767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7297136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7297311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7297686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7297875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7298120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7298363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7298798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7299203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7299434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7299661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7299916Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_j33kma8
2022-12-01T10:41:27.7300180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_j33kma8/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7300435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvup8n_7y
2022-12-01T10:41:27.7300703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvup8n_7y/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7300943Z [1669890743.017920] [7ff5451e42d4:30645:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7301166Z [1669890743.017920] [7ff5451e42d4:30645:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7301393Z [1669890743.015855] [7ff5451e42d4:30644:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7301623Z [1669890743.015855] [7ff5451e42d4:30644:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7301861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7302095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7302198Z ok (6.623s)
2022-12-01T10:41:27.7302266Z 
2022-12-01T10:41:27.7302546Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7302662Z Ran 1 test in 6.623s
2022-12-01T10:41:27.7302683Z 
2022-12-01T10:41:27.7302775Z OK
2022-12-01T10:41:27.7302795Z 
2022-12-01T10:41:27.7302901Z Generating XML reports...
2022-12-01T10:41:27.7303351Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103217.xml
2022-12-01T10:41:27.7303729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7303905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7304286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7304477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7304497Z 
2022-12-01T10:41:27.7304605Z Running tests...
2022-12-01T10:41:27.7304873Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7305169Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7305608Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T10:41:27.7305629Z 
2022-12-01T10:41:27.7305893Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7306009Z Ran 1 test in 0.004s
2022-12-01T10:41:27.7306029Z 
2022-12-01T10:41:27.7306137Z OK (skipped=1)
2022-12-01T10:41:27.7306157Z 
2022-12-01T10:41:27.7306282Z Generating XML reports...
2022-12-01T10:41:27.7306729Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103226.xml
2022-12-01T10:41:27.7307101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7307280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7307687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7307889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7307909Z 
2022-12-01T10:41:27.7308017Z Running tests...
2022-12-01T10:41:27.7308284Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7308595Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7309018Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7309039Z 
2022-12-01T10:41:27.7309297Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7309628Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7309656Z 
2022-12-01T10:41:27.7309771Z OK (skipped=1)
2022-12-01T10:41:27.7309790Z 
2022-12-01T10:41:27.7309900Z Generating XML reports...
2022-12-01T10:41:27.7310357Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103228.xml
2022-12-01T10:41:27.7310730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7310908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7311288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7311477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7311498Z 
2022-12-01T10:41:27.7311607Z Running tests...
2022-12-01T10:41:27.7311867Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7312177Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7312756Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7312796Z 
2022-12-01T10:41:27.7313039Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7313149Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7313169Z 
2022-12-01T10:41:27.7313276Z OK (skipped=1)
2022-12-01T10:41:27.7313295Z 
2022-12-01T10:41:27.7313419Z Generating XML reports...
2022-12-01T10:41:27.7313864Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103230.xml
2022-12-01T10:41:27.7314233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7314408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7314787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7314967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7314988Z 
2022-12-01T10:41:27.7315096Z Running tests...
2022-12-01T10:41:27.7315357Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7315664Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7316107Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7316130Z 
2022-12-01T10:41:27.7316390Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7316502Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7316522Z 
2022-12-01T10:41:27.7316630Z OK (skipped=1)
2022-12-01T10:41:27.7316649Z 
2022-12-01T10:41:27.7316774Z Generating XML reports...
2022-12-01T10:41:27.7317268Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103232.xml
2022-12-01T10:41:27.7317654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7317831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7318211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7318403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7318423Z 
2022-12-01T10:41:27.7318531Z Running tests...
2022-12-01T10:41:27.7318794Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7319104Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7319480Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-12-01T10:41:27.7319523Z 
2022-12-01T10:41:27.7319772Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7319884Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7319905Z 
2022-12-01T10:41:27.7320011Z OK (skipped=1)
2022-12-01T10:41:27.7320031Z 
2022-12-01T10:41:27.7320155Z Generating XML reports...
2022-12-01T10:41:27.7320603Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103234.xml
2022-12-01T10:41:27.7320978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7321157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7321542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7321715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7321802Z 
2022-12-01T10:41:27.7321898Z Running tests...
2022-12-01T10:41:27.7322166Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7322477Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7322740Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7322960Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31422
2022-12-01T10:41:27.7323178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31423
2022-12-01T10:41:27.7323552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7323731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7324096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7324296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7324668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7324842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7325221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7325410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7325654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7325897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7326277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7326724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7326968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7327194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7327450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2vd2dtb
2022-12-01T10:41:27.7327719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2vd2dtb/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7327971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq_bfx60
2022-12-01T10:41:27.7328236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq_bfx60/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7328470Z [1669890762.183442] [7ff5451e42d4:31423:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7328700Z [1669890762.183442] [7ff5451e42d4:31423:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7329481Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:41:27.7329716Z [1669890762.176489] [7ff5451e42d4:31422:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7329957Z [1669890762.176489] [7ff5451e42d4:31422:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7330796Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:41:27.7331040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7331275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7331376Z ok (6.412s)
2022-12-01T10:41:27.7331397Z 
2022-12-01T10:41:27.7331675Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7331793Z Ran 1 test in 6.412s
2022-12-01T10:41:27.7331813Z 
2022-12-01T10:41:27.7331909Z OK
2022-12-01T10:41:27.7331929Z 
2022-12-01T10:41:27.7332052Z Generating XML reports...
2022-12-01T10:41:27.7332504Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103237.xml
2022-12-01T10:41:27.7332861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7333039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7333421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7333615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7333635Z 
2022-12-01T10:41:27.7333744Z Running tests...
2022-12-01T10:41:27.7334008Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7334368Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7334658Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7335409Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.696s)
2022-12-01T10:41:27.7335432Z 
2022-12-01T10:41:27.7335698Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7335793Z Ran 1 test in 1.697s
2022-12-01T10:41:27.7335814Z 
2022-12-01T10:41:27.7335921Z OK (skipped=1)
2022-12-01T10:41:27.7335942Z 
2022-12-01T10:41:27.7336066Z Generating XML reports...
2022-12-01T10:41:27.7336514Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103245.xml
2022-12-01T10:41:27.7336893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7337069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7337449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7337639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7337659Z 
2022-12-01T10:41:27.7337749Z Running tests...
2022-12-01T10:41:27.7338013Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7338323Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7338809Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7338880Z 
2022-12-01T10:41:27.7339156Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7339271Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7339291Z 
2022-12-01T10:41:27.7339399Z OK (skipped=1)
2022-12-01T10:41:27.7339419Z 
2022-12-01T10:41:27.7339544Z Generating XML reports...
2022-12-01T10:41:27.7339988Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103249.xml
2022-12-01T10:41:27.7340342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7340518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7340897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7341086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7341110Z 
2022-12-01T10:41:27.7341217Z Running tests...
2022-12-01T10:41:27.7341481Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7341793Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7342215Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7342235Z 
2022-12-01T10:41:27.7342500Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7342596Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7342616Z 
2022-12-01T10:41:27.7342723Z OK (skipped=1)
2022-12-01T10:41:27.7342743Z 
2022-12-01T10:41:27.7342868Z Generating XML reports...
2022-12-01T10:41:27.7343311Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103251.xml
2022-12-01T10:41:27.7343682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7343916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7344303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7344492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7344514Z 
2022-12-01T10:41:27.7344621Z Running tests...
2022-12-01T10:41:27.7344863Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7345173Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7345438Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7345662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32057
2022-12-01T10:41:27.7345881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32058
2022-12-01T10:41:27.7346261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7346437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7346816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7346988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7347357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7347534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7347912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7348102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7348407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7348655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7349058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7349705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7349931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7350158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7350417Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp32falo97
2022-12-01T10:41:27.7350688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp32falo97/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7350951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwgkw1_e
2022-12-01T10:41:27.7351220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwgkw1_e/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7351457Z [1669890779.087883] [7ff5451e42d4:32058:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7351699Z [1669890779.087883] [7ff5451e42d4:32058:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7351927Z [1669890779.080538] [7ff5451e42d4:32057:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7352140Z [1669890779.080538] [7ff5451e42d4:32057:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7352243Z ok (6.099s)
2022-12-01T10:41:27.7352264Z 
2022-12-01T10:41:27.7352541Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7352660Z Ran 1 test in 6.099s
2022-12-01T10:41:27.7352750Z 
2022-12-01T10:41:27.7352849Z OK
2022-12-01T10:41:27.7352869Z 
2022-12-01T10:41:27.7352994Z Generating XML reports...
2022-12-01T10:41:27.7353449Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103253.xml
2022-12-01T10:41:27.7353822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7353982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7354362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7354554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7354575Z 
2022-12-01T10:41:27.7354682Z Running tests...
2022-12-01T10:41:27.7354948Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7355267Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7355547Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7356298Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.820s)
2022-12-01T10:41:27.7356320Z 
2022-12-01T10:41:27.7356585Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7356699Z Ran 1 test in 1.820s
2022-12-01T10:41:27.7356719Z 
2022-12-01T10:41:27.7356808Z OK (skipped=1)
2022-12-01T10:41:27.7356827Z 
2022-12-01T10:41:27.7356951Z Generating XML reports...
2022-12-01T10:41:27.7357402Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103302.xml
2022-12-01T10:41:27.7357852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7358034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7358413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7358604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7358624Z 
2022-12-01T10:41:27.7358731Z Running tests...
2022-12-01T10:41:27.7358992Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7359282Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7359709Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T10:41:27.7359734Z 
2022-12-01T10:41:27.7359997Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7360110Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7360130Z 
2022-12-01T10:41:27.7360238Z OK (skipped=1)
2022-12-01T10:41:27.7360257Z 
2022-12-01T10:41:27.7360380Z Generating XML reports...
2022-12-01T10:41:27.7360829Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103306.xml
2022-12-01T10:41:27.7361205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7361379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7361738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7361930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7361954Z 
2022-12-01T10:41:27.7362064Z Running tests...
2022-12-01T10:41:27.7362370Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7362694Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7362972Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7363716Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.764s)
2022-12-01T10:41:27.7363738Z 
2022-12-01T10:41:27.7363999Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7364111Z Ran 1 test in 1.765s
2022-12-01T10:41:27.7364132Z 
2022-12-01T10:41:27.7364223Z OK (skipped=1)
2022-12-01T10:41:27.7364265Z 
2022-12-01T10:41:27.7364372Z Generating XML reports...
2022-12-01T10:41:27.7364822Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103308.xml
2022-12-01T10:41:27.7365193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7365369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7365747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7365938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7365958Z 
2022-12-01T10:41:27.7366067Z Running tests...
2022-12-01T10:41:27.7366331Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7366624Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7366975Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7367196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32725
2022-12-01T10:41:27.7367414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32726
2022-12-01T10:41:27.7367789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7367966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7368346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7368538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7368886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7369064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7369443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7369633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7369878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7370122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7370523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7370915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7371144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7371359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7371666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygvus3fw
2022-12-01T10:41:27.7371945Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygvus3fw/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7372195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpij_sc5kw
2022-12-01T10:41:27.7372459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpij_sc5kw/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7372693Z [1669890797.539411] [7ff5451e42d4:32725:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7372937Z [1669890797.539411] [7ff5451e42d4:32725:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7373284Z STAGE:2022-12-01 10:33:18 32725:32725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7373522Z [1669890797.547234] [7ff5451e42d4:32726:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7373737Z [1669890797.547234] [7ff5451e42d4:32726:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7374071Z STAGE:2022-12-01 10:33:18 32726:32726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7374311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7374545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:41:27.7374883Z STAGE:2022-12-01 10:33:18 32725:32725 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7375214Z STAGE:2022-12-01 10:33:18 32726:32726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7375544Z STAGE:2022-12-01 10:33:18 32725:32725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7375933Z STAGE:2022-12-01 10:33:19 32725:32725 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7376036Z ok (7.709s)
2022-12-01T10:41:27.7376056Z 
2022-12-01T10:41:27.7376306Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7376418Z Ran 1 test in 7.709s
2022-12-01T10:41:27.7376438Z 
2022-12-01T10:41:27.7376531Z OK
2022-12-01T10:41:27.7376551Z 
2022-12-01T10:41:27.7376677Z Generating XML reports...
2022-12-01T10:41:27.7377125Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103312.xml
2022-12-01T10:41:27.7377497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7377671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7378052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7378250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7378271Z 
2022-12-01T10:41:27.7378364Z Running tests...
2022-12-01T10:41:27.7378633Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7378945Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7379210Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7379430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33106
2022-12-01T10:41:27.7379646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33107
2022-12-01T10:41:27.7380016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7380195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7380603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7380854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7381230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7381406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7381779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7381966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7382210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7382456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7382868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7383247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7383478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7383704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7383962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpog7yozgj
2022-12-01T10:41:27.7384232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpog7yozgj/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7384483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5lf4ts89
2022-12-01T10:41:27.7384750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5lf4ts89/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7385049Z [1669890807.282303] [7ff5451e42d4:33107:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7385292Z [1669890807.282303] [7ff5451e42d4:33107:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7385497Z [1669890807.274619] [7ff5451e42d4:33106:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7385726Z [1669890807.274619] [7ff5451e42d4:33106:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7385830Z ok (5.804s)
2022-12-01T10:41:27.7385850Z 
2022-12-01T10:41:27.7386124Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7386237Z Ran 1 test in 5.804s
2022-12-01T10:41:27.7386256Z 
2022-12-01T10:41:27.7386349Z OK
2022-12-01T10:41:27.7386368Z 
2022-12-01T10:41:27.7386497Z Generating XML reports...
2022-12-01T10:41:27.7386945Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103322.xml
2022-12-01T10:41:27.7387299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7387477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7387858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7388048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7388068Z 
2022-12-01T10:41:27.7388180Z Running tests...
2022-12-01T10:41:27.7388439Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7388749Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7389024Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7389296Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33420
2022-12-01T10:41:27.7389678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33421
2022-12-01T10:41:27.7390073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7390249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7390625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7390815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7391180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7391357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7391745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7391918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7392164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7392408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7392805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7393199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7393429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7393656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7394011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu7ue3d2f
2022-12-01T10:41:27.7394285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu7ue3d2f/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7394520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7rb4lw9b
2022-12-01T10:41:27.7394788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7rb4lw9b/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7395021Z [1669890815.540158] [7ff5451e42d4:33420:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7395260Z [1669890815.540158] [7ff5451e42d4:33420:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7395482Z [1669890815.546852] [7ff5451e42d4:33421:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7395722Z [1669890815.546852] [7ff5451e42d4:33421:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7395825Z ok (6.002s)
2022-12-01T10:41:27.7395847Z 
2022-12-01T10:41:27.7396119Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7396232Z Ran 1 test in 6.002s
2022-12-01T10:41:27.7396252Z 
2022-12-01T10:41:27.7396326Z OK
2022-12-01T10:41:27.7396365Z 
2022-12-01T10:41:27.7396470Z Generating XML reports...
2022-12-01T10:41:27.7396918Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103330.xml
2022-12-01T10:41:27.7397291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7397467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7397841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7398098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7398121Z 
2022-12-01T10:41:27.7398236Z Running tests...
2022-12-01T10:41:27.7398504Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7398796Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7399071Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7399817Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-12-01T10:41:27.7399838Z 
2022-12-01T10:41:27.7400104Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7400220Z Ran 1 test in 1.630s
2022-12-01T10:41:27.7400242Z 
2022-12-01T10:41:27.7400349Z OK (skipped=1)
2022-12-01T10:41:27.7400368Z 
2022-12-01T10:41:27.7400493Z Generating XML reports...
2022-12-01T10:41:27.7400937Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103338.xml
2022-12-01T10:41:27.7401307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7401488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7401852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7402047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7402067Z 
2022-12-01T10:41:27.7402174Z Running tests...
2022-12-01T10:41:27.7402436Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7402810Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7403071Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7403813Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.800s)
2022-12-01T10:41:27.7403834Z 
2022-12-01T10:41:27.7404100Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7404211Z Ran 1 test in 1.800s
2022-12-01T10:41:27.7404231Z 
2022-12-01T10:41:27.7404319Z OK (skipped=1)
2022-12-01T10:41:27.7404360Z 
2022-12-01T10:41:27.7404468Z Generating XML reports...
2022-12-01T10:41:27.7404915Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103342.xml
2022-12-01T10:41:27.7405292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7405469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7405852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7406042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7406063Z 
2022-12-01T10:41:27.7406171Z Running tests...
2022-12-01T10:41:27.7406433Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7406726Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7407016Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7407797Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.836s)
2022-12-01T10:41:27.7407820Z 
2022-12-01T10:41:27.7408090Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7408204Z Ran 1 test in 1.836s
2022-12-01T10:41:27.7408223Z 
2022-12-01T10:41:27.7408332Z OK (skipped=1)
2022-12-01T10:41:27.7408352Z 
2022-12-01T10:41:27.7408476Z Generating XML reports...
2022-12-01T10:41:27.7408918Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103346.xml
2022-12-01T10:41:27.7409288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7409470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7409835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7410027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7410047Z 
2022-12-01T10:41:27.7410154Z Running tests...
2022-12-01T10:41:27.7410414Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7410720Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7411165Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-12-01T10:41:27.7411185Z 
2022-12-01T10:41:27.7411449Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7411562Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7411582Z 
2022-12-01T10:41:27.7411723Z OK (skipped=1)
2022-12-01T10:41:27.7411763Z 
2022-12-01T10:41:27.7411869Z Generating XML reports...
2022-12-01T10:41:27.7412324Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103350.xml
2022-12-01T10:41:27.7412697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7412875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7413254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7413444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7413465Z 
2022-12-01T10:41:27.7413572Z Running tests...
2022-12-01T10:41:27.7413834Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7414128Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7414396Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7414614Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34251
2022-12-01T10:41:27.7414831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34252
2022-12-01T10:41:27.7415196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7415371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7415746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7415937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7416307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7416469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7416893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7417090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7417336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7417579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7417980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7418378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7418607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7418838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7419063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7419301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7419700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7420086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7420189Z ok (4.236s)
2022-12-01T10:41:27.7420210Z 
2022-12-01T10:41:27.7420473Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7420585Z Ran 1 test in 4.236s
2022-12-01T10:41:27.7420605Z 
2022-12-01T10:41:27.7420697Z OK
2022-12-01T10:41:27.7420716Z 
2022-12-01T10:41:27.7420822Z Generating XML reports...
2022-12-01T10:41:27.7421334Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103352.xml
2022-12-01T10:41:27.7421707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7421882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7422262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7422451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7422471Z 
2022-12-01T10:41:27.7422579Z Running tests...
2022-12-01T10:41:27.7422850Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7423160Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7423397Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7423624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34574
2022-12-01T10:41:27.7423844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34575
2022-12-01T10:41:27.7424216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7424392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7424772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7424963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7425331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7425487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7425945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7426144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7426390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7426633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7427033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7427434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7427664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7427900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7428112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7428345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7428743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7429134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7429238Z ok (4.167s)
2022-12-01T10:41:27.7429259Z 
2022-12-01T10:41:27.7429755Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7429873Z Ran 1 test in 4.167s
2022-12-01T10:41:27.7429895Z 
2022-12-01T10:41:27.7429992Z OK
2022-12-01T10:41:27.7430013Z 
2022-12-01T10:41:27.7430137Z Generating XML reports...
2022-12-01T10:41:27.7430575Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103359.xml
2022-12-01T10:41:27.7431041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7431219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7431599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7431792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7431812Z 
2022-12-01T10:41:27.7431920Z Running tests...
2022-12-01T10:41:27.7432183Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7432496Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7432754Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7433509Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2022-12-01T10:41:27.7433550Z 
2022-12-01T10:41:27.7433797Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7433909Z Ran 1 test in 1.644s
2022-12-01T10:41:27.7433929Z 
2022-12-01T10:41:27.7434038Z OK (skipped=1)
2022-12-01T10:41:27.7434057Z 
2022-12-01T10:41:27.7434180Z Generating XML reports...
2022-12-01T10:41:27.7434624Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103405.xml
2022-12-01T10:41:27.7434994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7435171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7435610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7435796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7435819Z 
2022-12-01T10:41:27.7435929Z Running tests...
2022-12-01T10:41:27.7436201Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7436515Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7436936Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7436958Z 
2022-12-01T10:41:27.7437220Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7437334Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7437356Z 
2022-12-01T10:41:27.7437464Z OK (skipped=1)
2022-12-01T10:41:27.7437491Z 
2022-12-01T10:41:27.7437615Z Generating XML reports...
2022-12-01T10:41:27.7438049Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103409.xml
2022-12-01T10:41:27.7438422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7438597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7438981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7439172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7439192Z 
2022-12-01T10:41:27.7439300Z Running tests...
2022-12-01T10:41:27.7439561Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7439871Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7440185Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7440411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35110
2022-12-01T10:41:27.7440631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35111
2022-12-01T10:41:27.7441006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7441184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7441563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7441752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7442113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7442288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7442652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7442841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7443087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7443330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7443732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7444130Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7444359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7444589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7444695Z ok (4.060s)
2022-12-01T10:41:27.7444757Z 
2022-12-01T10:41:27.7445015Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7445131Z Ran 1 test in 4.060s
2022-12-01T10:41:27.7445152Z 
2022-12-01T10:41:27.7445246Z OK
2022-12-01T10:41:27.7445268Z 
2022-12-01T10:41:27.7445392Z Generating XML reports...
2022-12-01T10:41:27.7445841Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103411.xml
2022-12-01T10:41:27.7446211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7446387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7446765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7446938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7446979Z 
2022-12-01T10:41:27.7447072Z Running tests...
2022-12-01T10:41:27.7447338Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7447648Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7447905Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7447926Z 
2022-12-01T10:41:27.7448185Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7448299Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7448319Z 
2022-12-01T10:41:27.7448428Z OK (skipped=1)
2022-12-01T10:41:27.7448447Z 
2022-12-01T10:41:27.7448570Z Generating XML reports...
2022-12-01T10:41:27.7448994Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103417.xml
2022-12-01T10:41:27.7449364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7449599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7449980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7450174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7450194Z 
2022-12-01T10:41:27.7450302Z Running tests...
2022-12-01T10:41:27.7450566Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7450875Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7451125Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7451163Z 
2022-12-01T10:41:27.7451407Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7451524Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7451544Z 
2022-12-01T10:41:27.7451651Z OK (skipped=1)
2022-12-01T10:41:27.7451673Z 
2022-12-01T10:41:27.7451796Z Generating XML reports...
2022-12-01T10:41:27.7452236Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103419.xml
2022-12-01T10:41:27.7452604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7452779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7453157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7453331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7453371Z 
2022-12-01T10:41:27.7453462Z Running tests...
2022-12-01T10:41:27.7453725Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7454038Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7454335Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:41:27.7454357Z 
2022-12-01T10:41:27.7454630Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7454743Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7454763Z 
2022-12-01T10:41:27.7454870Z OK (skipped=1)
2022-12-01T10:41:27.7454890Z 
2022-12-01T10:41:27.7455015Z Generating XML reports...
2022-12-01T10:41:27.7455442Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103422.xml
2022-12-01T10:41:27.7455812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7455988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7456363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7456562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7456582Z 
2022-12-01T10:41:27.7456692Z Running tests...
2022-12-01T10:41:27.7456955Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7457266Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7457536Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7457556Z 
2022-12-01T10:41:27.7457801Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7457913Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7457933Z 
2022-12-01T10:41:27.7458039Z OK (skipped=1)
2022-12-01T10:41:27.7458058Z 
2022-12-01T10:41:27.7458182Z Generating XML reports...
2022-12-01T10:41:27.7458630Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103424.xml
2022-12-01T10:41:27.7459082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7459259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7459638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7459810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7459848Z 
2022-12-01T10:41:27.7459938Z Running tests...
2022-12-01T10:41:27.7460200Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7460512Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7460780Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7460804Z 
2022-12-01T10:41:27.7461071Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7461184Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7461205Z 
2022-12-01T10:41:27.7461313Z OK (skipped=1)
2022-12-01T10:41:27.7461332Z 
2022-12-01T10:41:27.7461454Z Generating XML reports...
2022-12-01T10:41:27.7461883Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103426.xml
2022-12-01T10:41:27.7462259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7462432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7462808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7462999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7463022Z 
2022-12-01T10:41:27.7463130Z Running tests...
2022-12-01T10:41:27.7463442Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7463766Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7464155Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7464176Z 
2022-12-01T10:41:27.7464415Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7464527Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7464546Z 
2022-12-01T10:41:27.7464652Z OK (skipped=1)
2022-12-01T10:41:27.7464672Z 
2022-12-01T10:41:27.7464795Z Generating XML reports...
2022-12-01T10:41:27.7465239Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103428.xml
2022-12-01T10:41:27.7465609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7465793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7466175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7466366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7466387Z 
2022-12-01T10:41:27.7466478Z Running tests...
2022-12-01T10:41:27.7466741Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7467050Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7467456Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7467477Z 
2022-12-01T10:41:27.7467737Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7467849Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7467918Z 
2022-12-01T10:41:27.7468031Z OK (skipped=1)
2022-12-01T10:41:27.7468051Z 
2022-12-01T10:41:27.7468178Z Generating XML reports...
2022-12-01T10:41:27.7468607Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103430.xml
2022-12-01T10:41:27.7468973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7469149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7469764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7469963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7469983Z 
2022-12-01T10:41:27.7470091Z Running tests...
2022-12-01T10:41:27.7470358Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7470671Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7470928Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7471132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36031
2022-12-01T10:41:27.7471348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36032
2022-12-01T10:41:27.7471723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7471898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7472275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7472467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7472832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7473080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7473452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7473646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7473892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7474136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7474534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7474926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7475156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7475401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7475624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7475841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7476239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7476627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7476733Z ok (13.545s)
2022-12-01T10:41:27.7476753Z 
2022-12-01T10:41:27.7477015Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7477128Z Ran 1 test in 13.545s
2022-12-01T10:41:27.7477148Z 
2022-12-01T10:41:27.7477241Z OK
2022-12-01T10:41:27.7477321Z 
2022-12-01T10:41:27.7477453Z Generating XML reports...
2022-12-01T10:41:27.7477911Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103432.xml
2022-12-01T10:41:27.7478261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7478438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7478817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7479010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7479031Z 
2022-12-01T10:41:27.7479139Z Running tests...
2022-12-01T10:41:27.7479405Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7479718Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7479999Z   test_get_future (__main__.TestDistBackendWithSpawn) ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T10:41:27.7480020Z 
2022-12-01T10:41:27.7480280Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7480375Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7480395Z 
2022-12-01T10:41:27.7480502Z OK (skipped=1)
2022-12-01T10:41:27.7480521Z 
2022-12-01T10:41:27.7480644Z Generating XML reports...
2022-12-01T10:41:27.7481136Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103448.xml
2022-12-01T10:41:27.7481509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7481686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7482060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7482255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7482323Z 
2022-12-01T10:41:27.7482437Z Running tests...
2022-12-01T10:41:27.7482687Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7482997Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7483239Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7483458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36430
2022-12-01T10:41:27.7483674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36431
2022-12-01T10:41:27.7484046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7484221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7484600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7484784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7485154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7485330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7485706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7485898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7486144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7486388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7486788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7487246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7487459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7487688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7487791Z ok (4.233s)
2022-12-01T10:41:27.7487812Z 
2022-12-01T10:41:27.7488075Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7488189Z Ran 1 test in 4.233s
2022-12-01T10:41:27.7488211Z 
2022-12-01T10:41:27.7488305Z OK
2022-12-01T10:41:27.7488326Z 
2022-12-01T10:41:27.7488450Z Generating XML reports...
2022-12-01T10:41:27.7488895Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103450.xml
2022-12-01T10:41:27.7489250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7489434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7489818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7490010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7490031Z 
2022-12-01T10:41:27.7490140Z Running tests...
2022-12-01T10:41:27.7490406Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7490714Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7490982Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7491202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36738
2022-12-01T10:41:27.7491403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36739
2022-12-01T10:41:27.7491825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7492009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7492389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7492579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7492945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7493119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7493497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7493671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7493922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7494167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7494566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7494964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7495194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7495422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7495660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7495900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7496334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7496729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7496832Z ok (3.809s)
2022-12-01T10:41:27.7496852Z 
2022-12-01T10:41:27.7497114Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7497227Z Ran 1 test in 3.810s
2022-12-01T10:41:27.7497248Z 
2022-12-01T10:41:27.7497341Z OK
2022-12-01T10:41:27.7497362Z 
2022-12-01T10:41:27.7497485Z Generating XML reports...
2022-12-01T10:41:27.7497929Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103456.xml
2022-12-01T10:41:27.7498300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7498463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7498844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7499034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7499054Z 
2022-12-01T10:41:27.7499162Z Running tests...
2022-12-01T10:41:27.7499428Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7499738Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7499998Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7500220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36971
2022-12-01T10:41:27.7500420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36972
2022-12-01T10:41:27.7500838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7501020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7501405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7501595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7501963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7502137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7502511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7502701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7502928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7503181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7503582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7503978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7504205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7504445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7504669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7504904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7505358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7505731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7505834Z ok (4.129s)
2022-12-01T10:41:27.7505855Z 
2022-12-01T10:41:27.7506118Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7506229Z Ran 1 test in 4.130s
2022-12-01T10:41:27.7506248Z 
2022-12-01T10:41:27.7506341Z OK
2022-12-01T10:41:27.7506360Z 
2022-12-01T10:41:27.7506485Z Generating XML reports...
2022-12-01T10:41:27.7506935Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103502.xml
2022-12-01T10:41:27.7507307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7507465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7507851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7508044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7508065Z 
2022-12-01T10:41:27.7508172Z Running tests...
2022-12-01T10:41:27.7508433Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7508743Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7509145Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T10:41:27.7509166Z 
2022-12-01T10:41:27.7509642Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7509766Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7509787Z 
2022-12-01T10:41:27.7509878Z OK (skipped=1)
2022-12-01T10:41:27.7509898Z 
2022-12-01T10:41:27.7510026Z Generating XML reports...
2022-12-01T10:41:27.7510550Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103509.xml
2022-12-01T10:41:27.7510932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7511107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7511486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7511680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7511701Z 
2022-12-01T10:41:27.7511807Z Running tests...
2022-12-01T10:41:27.7512070Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7512362Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7512601Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7512829Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37335
2022-12-01T10:41:27.7513049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37336
2022-12-01T10:41:27.7513421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7513595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7513976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7514166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7514515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7514690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7515141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7515330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7515573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7515821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7516219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7516614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7516845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7517057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7517297Z [1669890916.779737] [7ff5451e42d4:37336:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7517538Z [1669890916.779737] [7ff5451e42d4:37336:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7517762Z [1669890916.766188] [7ff5451e42d4:37335:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7517997Z [1669890916.766188] [7ff5451e42d4:37335:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7518099Z ok (6.080s)
2022-12-01T10:41:27.7518119Z 
2022-12-01T10:41:27.7518387Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7518501Z Ran 1 test in 6.080s
2022-12-01T10:41:27.7518521Z 
2022-12-01T10:41:27.7518614Z OK
2022-12-01T10:41:27.7518635Z 
2022-12-01T10:41:27.7518739Z Generating XML reports...
2022-12-01T10:41:27.7519238Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103511.xml
2022-12-01T10:41:27.7519621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7519797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7520175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7520367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7520387Z 
2022-12-01T10:41:27.7520496Z Running tests...
2022-12-01T10:41:27.7520762Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7521056Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7521297Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7521525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37703
2022-12-01T10:41:27.7521742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37704
2022-12-01T10:41:27.7522114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7522289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7522667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7522856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7523221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7523378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7523756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7524012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7524257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7524499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7524900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7525297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7525525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7525753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7525977Z [1669890925.047234] [7ff5451e42d4:37704:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7526217Z [1669890925.047234] [7ff5451e42d4:37704:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7526443Z [1669890925.006829] [7ff5451e42d4:37703:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7526671Z [1669890925.006829] [7ff5451e42d4:37703:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7526774Z ok (5.947s)
2022-12-01T10:41:27.7526795Z 
2022-12-01T10:41:27.7527062Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7527176Z Ran 1 test in 5.947s
2022-12-01T10:41:27.7527196Z 
2022-12-01T10:41:27.7527289Z OK
2022-12-01T10:41:27.7527309Z 
2022-12-01T10:41:27.7527431Z Generating XML reports...
2022-12-01T10:41:27.7527906Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103519.xml
2022-12-01T10:41:27.7528288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7528467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7528845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7529035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7529056Z 
2022-12-01T10:41:27.7529164Z Running tests...
2022-12-01T10:41:27.7529425Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7529738Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7529989Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7530219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38029
2022-12-01T10:41:27.7530436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38030
2022-12-01T10:41:27.7530807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7530981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7531356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7531545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7531911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7532086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7532449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7532697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7532943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7533187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7533590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7533985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7534216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7534554Z STAGE:2022-12-01 10:35:31 38029:38029 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7534787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7535109Z STAGE:2022-12-01 10:35:31 38030:38030 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7535346Z [1669890933.575091] [7ff5451e42d4:38030:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7535586Z [1669890933.575091] [7ff5451e42d4:38030:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7535924Z STAGE:2022-12-01 10:35:34 38030:38030 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7536152Z [1669890933.612704] [7ff5451e42d4:38029:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7536381Z [1669890933.612704] [7ff5451e42d4:38029:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7536762Z STAGE:2022-12-01 10:35:34 38029:38029 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7536872Z ok (7.101s)
2022-12-01T10:41:27.7536893Z 
2022-12-01T10:41:27.7537161Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7537257Z Ran 1 test in 7.101s
2022-12-01T10:41:27.7537277Z 
2022-12-01T10:41:27.7537370Z OK
2022-12-01T10:41:27.7537390Z 
2022-12-01T10:41:27.7537514Z Generating XML reports...
2022-12-01T10:41:27.7537961Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103528.xml
2022-12-01T10:41:27.7538329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7538505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7538883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7539082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7539105Z 
2022-12-01T10:41:27.7539214Z Running tests...
2022-12-01T10:41:27.7539461Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7539774Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7540036Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7540254Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38413
2022-12-01T10:41:27.7540472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38414
2022-12-01T10:41:27.7540839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7541015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7541452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7541625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7541996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7542171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7542548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7542736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7542983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7543228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7543630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7544032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7544246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7544584Z STAGE:2022-12-01 10:35:41 38413:38413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7544812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7545144Z STAGE:2022-12-01 10:35:41 38414:38414 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7545378Z [1669890943.006565] [7ff5451e42d4:38413:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7545619Z [1669890943.006565] [7ff5451e42d4:38413:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7546023Z STAGE:2022-12-01 10:35:43 38413:38413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7546262Z [1669890943.051054] [7ff5451e42d4:38414:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7546496Z [1669890943.051054] [7ff5451e42d4:38414:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7546817Z STAGE:2022-12-01 10:35:43 38414:38414 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7546919Z ok (7.157s)
2022-12-01T10:41:27.7546940Z 
2022-12-01T10:41:27.7547209Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7547322Z Ran 1 test in 7.157s
2022-12-01T10:41:27.7547343Z 
2022-12-01T10:41:27.7547435Z OK
2022-12-01T10:41:27.7547455Z 
2022-12-01T10:41:27.7547577Z Generating XML reports...
2022-12-01T10:41:27.7548032Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103537.xml
2022-12-01T10:41:27.7548404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7548563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7548942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7549133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7549153Z 
2022-12-01T10:41:27.7549262Z Running tests...
2022-12-01T10:41:27.7549747Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7550076Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7550505Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7550600Z 
2022-12-01T10:41:27.7550875Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7550990Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7551010Z 
2022-12-01T10:41:27.7551100Z OK (skipped=1)
2022-12-01T10:41:27.7551138Z 
2022-12-01T10:41:27.7551243Z Generating XML reports...
2022-12-01T10:41:27.7551690Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103546.xml
2022-12-01T10:41:27.7552060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7552235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7552615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7552807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7552832Z 
2022-12-01T10:41:27.7552939Z Running tests...
2022-12-01T10:41:27.7553205Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7553498Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7553949Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7553971Z 
2022-12-01T10:41:27.7554231Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7554344Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7554364Z 
2022-12-01T10:41:27.7554470Z OK (skipped=1)
2022-12-01T10:41:27.7554490Z 
2022-12-01T10:41:27.7554616Z Generating XML reports...
2022-12-01T10:41:27.7555065Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103549.xml
2022-12-01T10:41:27.7555498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7555683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7556044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7556235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7556255Z 
2022-12-01T10:41:27.7556362Z Running tests...
2022-12-01T10:41:27.7556624Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7556935Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7557343Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.7557364Z 
2022-12-01T10:41:27.7557622Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7557738Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7557758Z 
2022-12-01T10:41:27.7557868Z OK (skipped=1)
2022-12-01T10:41:27.7557888Z 
2022-12-01T10:41:27.7557992Z Generating XML reports...
2022-12-01T10:41:27.7558437Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103551.xml
2022-12-01T10:41:27.7558810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7558988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7559368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7559557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7559577Z 
2022-12-01T10:41:27.7559684Z Running tests...
2022-12-01T10:41:27.7559943Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7560295Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7560687Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.7560707Z 
2022-12-01T10:41:27.7560965Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7561077Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7561097Z 
2022-12-01T10:41:27.7561204Z OK (skipped=1)
2022-12-01T10:41:27.7561223Z 
2022-12-01T10:41:27.7561346Z Generating XML reports...
2022-12-01T10:41:27.7561788Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103553.xml
2022-12-01T10:41:27.7562157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7562335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7562700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7562891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7562911Z 
2022-12-01T10:41:27.7563018Z Running tests...
2022-12-01T10:41:27.7563281Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7563595Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7564015Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.7564035Z 
2022-12-01T10:41:27.7564295Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7564408Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7564427Z 
2022-12-01T10:41:27.7564533Z OK (skipped=1)
2022-12-01T10:41:27.7564556Z 
2022-12-01T10:41:27.7564662Z Generating XML reports...
2022-12-01T10:41:27.7565152Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103555.xml
2022-12-01T10:41:27.7565537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7565712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7566090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7566280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7566300Z 
2022-12-01T10:41:27.7566407Z Running tests...
2022-12-01T10:41:27.7566668Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7566976Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7567373Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.7567394Z 
2022-12-01T10:41:27.7567657Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7567770Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7567791Z 
2022-12-01T10:41:27.7567899Z OK (skipped=1)
2022-12-01T10:41:27.7567920Z 
2022-12-01T10:41:27.7568044Z Generating XML reports...
2022-12-01T10:41:27.7568487Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103557.xml
2022-12-01T10:41:27.7568856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7569030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7569407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7569654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7569678Z 
2022-12-01T10:41:27.7569786Z Running tests...
2022-12-01T10:41:27.7570052Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7570363Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7570770Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.7570791Z 
2022-12-01T10:41:27.7571049Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7571161Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7571181Z 
2022-12-01T10:41:27.7571289Z OK (skipped=1)
2022-12-01T10:41:27.7571308Z 
2022-12-01T10:41:27.7571431Z Generating XML reports...
2022-12-01T10:41:27.7571857Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103559.xml
2022-12-01T10:41:27.7572232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7572406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7572781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7572978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7572998Z 
2022-12-01T10:41:27.7573105Z Running tests...
2022-12-01T10:41:27.7573363Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7573670Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7574049Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:41:27.7574088Z 
2022-12-01T10:41:27.7574334Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7574491Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7574514Z 
2022-12-01T10:41:27.7574626Z OK (skipped=1)
2022-12-01T10:41:27.7574645Z 
2022-12-01T10:41:27.7574768Z Generating XML reports...
2022-12-01T10:41:27.7575218Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103601.xml
2022-12-01T10:41:27.7575584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7575760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7576138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7576311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7576349Z 
2022-12-01T10:41:27.7576440Z Running tests...
2022-12-01T10:41:27.7576709Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7577025Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7577425Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:41:27.7577445Z 
2022-12-01T10:41:27.7577706Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7577819Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7577839Z 
2022-12-01T10:41:27.7577947Z OK (skipped=1)
2022-12-01T10:41:27.7577967Z 
2022-12-01T10:41:27.7578092Z Generating XML reports...
2022-12-01T10:41:27.7578519Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103603.xml
2022-12-01T10:41:27.7578891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7579119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7579505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7579697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7579718Z 
2022-12-01T10:41:27.7579825Z Running tests...
2022-12-01T10:41:27.7580087Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7580398Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7580782Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:41:27.7580819Z 
2022-12-01T10:41:27.7581117Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7581229Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7581249Z 
2022-12-01T10:41:27.7581358Z OK (skipped=1)
2022-12-01T10:41:27.7581377Z 
2022-12-01T10:41:27.7581499Z Generating XML reports...
2022-12-01T10:41:27.7581947Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103605.xml
2022-12-01T10:41:27.7582317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7582490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7582869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7583042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7583080Z 
2022-12-01T10:41:27.7583171Z Running tests...
2022-12-01T10:41:27.7583432Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7583743Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7584188Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-12-01T10:41:27.7584211Z 
2022-12-01T10:41:27.7584480Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7584592Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7584611Z 
2022-12-01T10:41:27.7584718Z OK (skipped=1)
2022-12-01T10:41:27.7584737Z 
2022-12-01T10:41:27.7584860Z Generating XML reports...
2022-12-01T10:41:27.7585285Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103607.xml
2022-12-01T10:41:27.7585659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7585839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7586217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7586414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7586434Z 
2022-12-01T10:41:27.7586540Z Running tests...
2022-12-01T10:41:27.7586801Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7587110Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7587407Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-12-01T10:41:27.7587427Z 
2022-12-01T10:41:27.7587675Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7587786Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7587806Z 
2022-12-01T10:41:27.7587912Z OK (skipped=1)
2022-12-01T10:41:27.7587931Z 
2022-12-01T10:41:27.7588053Z Generating XML reports...
2022-12-01T10:41:27.7588496Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103609.xml
2022-12-01T10:41:27.7588941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7589117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7589725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7589923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7589945Z 
2022-12-01T10:41:27.7590037Z Running tests...
2022-12-01T10:41:27.7590308Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7590622Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7590873Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:41:27.7590893Z 
2022-12-01T10:41:27.7591168Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7591284Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7591305Z 
2022-12-01T10:41:27.7591412Z OK (skipped=1)
2022-12-01T10:41:27.7591431Z 
2022-12-01T10:41:27.7591554Z Generating XML reports...
2022-12-01T10:41:27.7591979Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103611.xml
2022-12-01T10:41:27.7592349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7592526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7592904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7593096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7593116Z 
2022-12-01T10:41:27.7593223Z Running tests...
2022-12-01T10:41:27.7593488Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7593868Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7594152Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:41:27.7594174Z 
2022-12-01T10:41:27.7594420Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7594533Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7594553Z 
2022-12-01T10:41:27.7594659Z OK (skipped=1)
2022-12-01T10:41:27.7594678Z 
2022-12-01T10:41:27.7594803Z Generating XML reports...
2022-12-01T10:41:27.7595246Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103614.xml
2022-12-01T10:41:27.7595616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7595794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7596177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7596367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7596388Z 
2022-12-01T10:41:27.7596479Z Running tests...
2022-12-01T10:41:27.7596740Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7597051Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7597359Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:41:27.7597379Z 
2022-12-01T10:41:27.7597638Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7597749Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7597769Z 
2022-12-01T10:41:27.7597940Z OK (skipped=1)
2022-12-01T10:41:27.7597959Z 
2022-12-01T10:41:27.7598083Z Generating XML reports...
2022-12-01T10:41:27.7598534Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103616.xml
2022-12-01T10:41:27.7598885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7599062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7599439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7599630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7599651Z 
2022-12-01T10:41:27.7599760Z Running tests...
2022-12-01T10:41:27.7600019Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7600331Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7600638Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7600843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40170
2022-12-01T10:41:27.7601063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40171
2022-12-01T10:41:27.7601430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7601604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7601980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7602169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7602536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7602715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7603136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7603316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7603561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7603806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7604207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7604602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7604832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7605068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7605171Z ok (4.096s)
2022-12-01T10:41:27.7605191Z 
2022-12-01T10:41:27.7605457Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7605552Z Ran 1 test in 4.097s
2022-12-01T10:41:27.7605571Z 
2022-12-01T10:41:27.7605663Z OK
2022-12-01T10:41:27.7605683Z 
2022-12-01T10:41:27.7605806Z Generating XML reports...
2022-12-01T10:41:27.7606253Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103618.xml
2022-12-01T10:41:27.7606621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7606794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7607168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7607418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7607439Z 
2022-12-01T10:41:27.7607529Z Running tests...
2022-12-01T10:41:27.7607795Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7608106Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7608401Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7608622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40396
2022-12-01T10:41:27.7608840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40397
2022-12-01T10:41:27.7609214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7609389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7609776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7609950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7610320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7610494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7610869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7611057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7611300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7611546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7611999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7612391Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7612621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7612847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7612950Z ok (4.109s)
2022-12-01T10:41:27.7612971Z 
2022-12-01T10:41:27.7613236Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7613347Z Ran 1 test in 4.109s
2022-12-01T10:41:27.7613366Z 
2022-12-01T10:41:27.7613459Z OK
2022-12-01T10:41:27.7613478Z 
2022-12-01T10:41:27.7613601Z Generating XML reports...
2022-12-01T10:41:27.7614050Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103624.xml
2022-12-01T10:41:27.7614413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7614589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7614970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7615161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7615181Z 
2022-12-01T10:41:27.7615289Z Running tests...
2022-12-01T10:41:27.7615551Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7615862Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7616139Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:41:27.7616159Z 
2022-12-01T10:41:27.7616473Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7616574Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7616594Z 
2022-12-01T10:41:27.7616704Z OK (skipped=1)
2022-12-01T10:41:27.7616723Z 
2022-12-01T10:41:27.7616848Z Generating XML reports...
2022-12-01T10:41:27.7617290Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103630.xml
2022-12-01T10:41:27.7617657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7617832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7618208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7618399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7618419Z 
2022-12-01T10:41:27.7618512Z Running tests...
2022-12-01T10:41:27.7618776Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7619090Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7619391Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2022-12-01T10:41:27.7619412Z 
2022-12-01T10:41:27.7619673Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7619786Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7619805Z 
2022-12-01T10:41:27.7619916Z OK (skipped=1)
2022-12-01T10:41:27.7619936Z 
2022-12-01T10:41:27.7620058Z Generating XML reports...
2022-12-01T10:41:27.7620503Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103632.xml
2022-12-01T10:41:27.7620859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7621039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7621463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7621662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7621683Z 
2022-12-01T10:41:27.7621791Z Running tests...
2022-12-01T10:41:27.7622053Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7622361Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7622638Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7623387Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.709s)
2022-12-01T10:41:27.7623416Z 
2022-12-01T10:41:27.7623683Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7623776Z Ran 1 test in 1.710s
2022-12-01T10:41:27.7623797Z 
2022-12-01T10:41:27.7623904Z OK (skipped=1)
2022-12-01T10:41:27.7623924Z 
2022-12-01T10:41:27.7624047Z Generating XML reports...
2022-12-01T10:41:27.7624492Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103635.xml
2022-12-01T10:41:27.7624862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7625037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7625413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7625605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7625675Z 
2022-12-01T10:41:27.7625790Z Running tests...
2022-12-01T10:41:27.7626043Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7626351Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7626636Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7626854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40984
2022-12-01T10:41:27.7627069Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40985
2022-12-01T10:41:27.7627436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7627611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7627986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7628165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7628534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7628708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7629087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7629276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7629699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7629951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7630357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7630831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7631058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7631287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7631546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5a3l510
2022-12-01T10:41:27.7631817Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5a3l510/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7632074Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfawbb3k
2022-12-01T10:41:27.7632342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfawbb3k/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7632577Z [1669891004.293952] [7ff5451e42d4:40984:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7632824Z [1669891004.293952] [7ff5451e42d4:40984:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7633050Z [1669891004.294235] [7ff5451e42d4:40985:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7633263Z [1669891004.294235] [7ff5451e42d4:40985:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7633367Z ok (6.355s)
2022-12-01T10:41:27.7633388Z 
2022-12-01T10:41:27.7633664Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7633779Z Ran 1 test in 6.356s
2022-12-01T10:41:27.7633798Z 
2022-12-01T10:41:27.7633890Z OK
2022-12-01T10:41:27.7633910Z 
2022-12-01T10:41:27.7634034Z Generating XML reports...
2022-12-01T10:41:27.7634477Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103639.xml
2022-12-01T10:41:27.7634925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7635083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7635462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7635653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7635674Z 
2022-12-01T10:41:27.7635781Z Running tests...
2022-12-01T10:41:27.7636047Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7636356Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7636625Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7636845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41325
2022-12-01T10:41:27.7637067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41326
2022-12-01T10:41:27.7637422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7637599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7637971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7638147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7638530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7638773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7639157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7639400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7639633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7640037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7640281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7640678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7640906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7641132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7641366Z [1669891013.873699] [7ff5451e42d4:41325:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7641613Z [1669891013.873699] [7ff5451e42d4:41325:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7641837Z [1669891013.878826] [7ff5451e42d4:41326:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7642065Z [1669891013.878826] [7ff5451e42d4:41326:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7642150Z ok (6.898s)
2022-12-01T10:41:27.7642170Z 
2022-12-01T10:41:27.7642437Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7642550Z Ran 1 test in 6.898s
2022-12-01T10:41:27.7642570Z 
2022-12-01T10:41:27.7642663Z OK
2022-12-01T10:41:27.7642682Z 
2022-12-01T10:41:27.7642807Z Generating XML reports...
2022-12-01T10:41:27.7643252Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103647.xml
2022-12-01T10:41:27.7643682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7643860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7644223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7644416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7644436Z 
2022-12-01T10:41:27.7644545Z Running tests...
2022-12-01T10:41:27.7644807Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7645117Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7645403Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7645622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41591
2022-12-01T10:41:27.7645844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41592
2022-12-01T10:41:27.7646217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7646375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7646755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7646945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7647313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7647491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7647870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7648107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7648359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7648585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7648987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7649379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7649608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7649835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7650068Z [1669891023.008371] [7ff5451e42d4:41591:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7650316Z [1669891023.008371] [7ff5451e42d4:41591:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7650541Z [1669891023.013820] [7ff5451e42d4:41592:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7650769Z [1669891023.013820] [7ff5451e42d4:41592:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7650871Z ok (7.095s)
2022-12-01T10:41:27.7650891Z 
2022-12-01T10:41:27.7651143Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7651257Z Ran 1 test in 7.095s
2022-12-01T10:41:27.7651277Z 
2022-12-01T10:41:27.7651370Z OK
2022-12-01T10:41:27.7651389Z 
2022-12-01T10:41:27.7651511Z Generating XML reports...
2022-12-01T10:41:27.7651958Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103656.xml
2022-12-01T10:41:27.7652404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7652579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7652962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7653137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7653175Z 
2022-12-01T10:41:27.7653266Z Running tests...
2022-12-01T10:41:27.7653531Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7653843Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7654121Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7654871Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.686s)
2022-12-01T10:41:27.7654895Z 
2022-12-01T10:41:27.7655160Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7655273Z Ran 1 test in 1.686s
2022-12-01T10:41:27.7655293Z 
2022-12-01T10:41:27.7655401Z OK (skipped=1)
2022-12-01T10:41:27.7655421Z 
2022-12-01T10:41:27.7655546Z Generating XML reports...
2022-12-01T10:41:27.7655972Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103706.xml
2022-12-01T10:41:27.7656344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7656519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7656945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7657146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7657167Z 
2022-12-01T10:41:27.7657276Z Running tests...
2022-12-01T10:41:27.7657543Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7657853Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7658147Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7658869Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.694s)
2022-12-01T10:41:27.7658913Z 
2022-12-01T10:41:27.7659166Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7659279Z Ran 1 test in 1.694s
2022-12-01T10:41:27.7659299Z 
2022-12-01T10:41:27.7659409Z OK (skipped=1)
2022-12-01T10:41:27.7659429Z 
2022-12-01T10:41:27.7659551Z Generating XML reports...
2022-12-01T10:41:27.7659998Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103710.xml
2022-12-01T10:41:27.7660370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7660546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7660922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7661094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7661182Z 
2022-12-01T10:41:27.7661277Z Running tests...
2022-12-01T10:41:27.7661541Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7661853Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7662165Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7662385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42219
2022-12-01T10:41:27.7662603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42220
2022-12-01T10:41:27.7662973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7663147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7663510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7663709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7664081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7664255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7664630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7664819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7665062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7665306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7665687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7666132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7666372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7666599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7666750Z skip: Need at least 4 CUDA devices (3.919s)
2022-12-01T10:41:27.7666771Z 
2022-12-01T10:41:27.7667044Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7667157Z Ran 1 test in 3.919s
2022-12-01T10:41:27.7667177Z 
2022-12-01T10:41:27.7667284Z OK (skipped=1)
2022-12-01T10:41:27.7667304Z 
2022-12-01T10:41:27.7667428Z Generating XML reports...
2022-12-01T10:41:27.7667856Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103714.xml
2022-12-01T10:41:27.7668233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7668415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7668796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7668987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7669007Z 
2022-12-01T10:41:27.7669116Z Running tests...
2022-12-01T10:41:27.7669558Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7669891Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7670217Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7670422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42474
2022-12-01T10:41:27.7670731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42475
2022-12-01T10:41:27.7671113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7671289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7671665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7671855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7672221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7672396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7672756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7672945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7673195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7673438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7673837Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7674228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7674458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7674686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7674837Z skip: Need at least 4 CUDA devices (4.040s)
2022-12-01T10:41:27.7674858Z 
2022-12-01T10:41:27.7675105Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7675222Z Ran 1 test in 4.040s
2022-12-01T10:41:27.7675242Z 
2022-12-01T10:41:27.7675410Z OK (skipped=1)
2022-12-01T10:41:27.7675432Z 
2022-12-01T10:41:27.7675565Z Generating XML reports...
2022-12-01T10:41:27.7676015Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103720.xml
2022-12-01T10:41:27.7676382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7676559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7676932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7677124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7677145Z 
2022-12-01T10:41:27.7677235Z Running tests...
2022-12-01T10:41:27.7677496Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7677817Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7678101Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7678846Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.701s)
2022-12-01T10:41:27.7678867Z 
2022-12-01T10:41:27.7679133Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7679244Z Ran 1 test in 1.701s
2022-12-01T10:41:27.7679265Z 
2022-12-01T10:41:27.7679372Z OK (skipped=1)
2022-12-01T10:41:27.7679390Z 
2022-12-01T10:41:27.7679515Z Generating XML reports...
2022-12-01T10:41:27.7679961Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103726.xml
2022-12-01T10:41:27.7680381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7680557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7680985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7681180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7681200Z 
2022-12-01T10:41:27.7681309Z Running tests...
2022-12-01T10:41:27.7681571Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7681882Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7682147Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7682353Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42931
2022-12-01T10:41:27.7682573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42932
2022-12-01T10:41:27.7682948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7683126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7683501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7683690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7684053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7684226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7684603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7684830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7685084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7685330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7685734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7686128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7686356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7686596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7686819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7687041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7687436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7687774Z STAGE:2022-12-01 10:37:34 42932:42932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7688164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7688495Z STAGE:2022-12-01 10:37:34 42931:42931 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7688732Z [1669891056.107553] [7ff5451e42d4:42932:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7688971Z [1669891056.107553] [7ff5451e42d4:42932:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7689252Z [1669891056.107576] [7ff5451e42d4:42931:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7689483Z [1669891056.107576] [7ff5451e42d4:42931:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7690033Z STAGE:2022-12-01 10:37:36 42932:42932 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:37:36 42931:42931 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7690055Z 
2022-12-01T10:41:27.7690380Z STAGE:2022-12-01 10:37:36 42931:42931 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7690697Z STAGE:2022-12-01 10:37:36 42931:42931 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7691019Z STAGE:2022-12-01 10:37:36 42932:42932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7691356Z STAGE:2022-12-01 10:37:36 42932:42932 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7691460Z ok (7.116s)
2022-12-01T10:41:27.7691480Z 
2022-12-01T10:41:27.7691746Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7691862Z Ran 1 test in 7.117s
2022-12-01T10:41:27.7691882Z 
2022-12-01T10:41:27.7691975Z OK
2022-12-01T10:41:27.7691994Z 
2022-12-01T10:41:27.7692117Z Generating XML reports...
2022-12-01T10:41:27.7692545Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103730.xml
2022-12-01T10:41:27.7692915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7693091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7693470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7693715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7693736Z 
2022-12-01T10:41:27.7693849Z Running tests...
2022-12-01T10:41:27.7694114Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7694426Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7694690Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7694893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43234
2022-12-01T10:41:27.7695108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43235
2022-12-01T10:41:27.7695479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7695653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7696038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7696228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7696593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7696767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7697127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7697318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7697567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7697810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7698212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7698664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7698895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7699140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7699366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7699584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7699979Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7700369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7700713Z STAGE:2022-12-01 10:37:43 43234:43234 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7701041Z STAGE:2022-12-01 10:37:43 43235:43235 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7701276Z [1669891065.496298] [7ff5451e42d4:43235:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7701513Z [1669891065.496298] [7ff5451e42d4:43235:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7701736Z [1669891065.436538] [7ff5451e42d4:43234:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7701965Z [1669891065.436538] [7ff5451e42d4:43234:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7702558Z STAGE:2022-12-01 10:37:45 43235:43235 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:37:45 43234:43234 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7702584Z 
2022-12-01T10:41:27.7702924Z STAGE:2022-12-01 10:37:46 43235:43235 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7703229Z STAGE:2022-12-01 10:37:46 43234:43234 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7703765Z STAGE:2022-12-01 10:37:46 43235:43235 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:37:46 43234:43234 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7703786Z 
2022-12-01T10:41:27.7703888Z ok (7.132s)
2022-12-01T10:41:27.7703908Z 
2022-12-01T10:41:27.7704173Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7704284Z Ran 1 test in 7.132s
2022-12-01T10:41:27.7704304Z 
2022-12-01T10:41:27.7704396Z OK
2022-12-01T10:41:27.7704415Z 
2022-12-01T10:41:27.7704542Z Generating XML reports...
2022-12-01T10:41:27.7704991Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103740.xml
2022-12-01T10:41:27.7705367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7705525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7705904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7706094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7706114Z 
2022-12-01T10:41:27.7706223Z Running tests...
2022-12-01T10:41:27.7706490Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7706803Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7707078Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7707361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43587
2022-12-01T10:41:27.7707561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43588
2022-12-01T10:41:27.7707935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7708112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7708491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7708682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7709050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7709224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7709799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7709993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7710223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7710469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7710871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7711269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7711499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7711739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7712066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7712312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7712709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7713027Z STAGE:2022-12-01 10:37:53 43588:43588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7713414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7713743Z STAGE:2022-12-01 10:37:53 43587:43587 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7713982Z [1669891074.799174] [7ff5451e42d4:43587:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7714230Z [1669891074.799174] [7ff5451e42d4:43587:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7714455Z [1669891074.844129] [7ff5451e42d4:43588:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7714684Z [1669891074.844129] [7ff5451e42d4:43588:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7715235Z STAGE:2022-12-01 10:37:55 43587:43587 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:37:55 43588:43588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7715257Z 
2022-12-01T10:41:27.7715589Z STAGE:2022-12-01 10:37:55 43588:43588 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7715907Z STAGE:2022-12-01 10:37:55 43587:43587 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7716311Z STAGE:2022-12-01 10:37:55 43588:43588 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7716643Z STAGE:2022-12-01 10:37:55 43587:43587 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7716746Z ok (7.068s)
2022-12-01T10:41:27.7716766Z 
2022-12-01T10:41:27.7717031Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7717143Z Ran 1 test in 7.068s
2022-12-01T10:41:27.7717163Z 
2022-12-01T10:41:27.7717256Z OK
2022-12-01T10:41:27.7717275Z 
2022-12-01T10:41:27.7717399Z Generating XML reports...
2022-12-01T10:41:27.7717848Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103749.xml
2022-12-01T10:41:27.7718221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7718380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7718768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7718961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7718982Z 
2022-12-01T10:41:27.7719093Z Running tests...
2022-12-01T10:41:27.7719358Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7719671Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7719933Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7720153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43984
2022-12-01T10:41:27.7720351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43985
2022-12-01T10:41:27.7720718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7720895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7721317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7721516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7721885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7722060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7722437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7722630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7722856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7723097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7723505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7723904Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7724132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7724371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:41:27.7724595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7724828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:41:27.7725222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7725674Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:41:27.7726211Z STAGE:2022-12-01 10:38:02 43985:43985 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:38:02 43984:43984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7726233Z 
2022-12-01T10:41:27.7726468Z [1669891084.081485] [7ff5451e42d4:43985:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7726706Z [1669891084.081485] [7ff5451e42d4:43985:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7726930Z [1669891084.113137] [7ff5451e42d4:43984:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7727158Z [1669891084.113137] [7ff5451e42d4:43984:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7727710Z STAGE:2022-12-01 10:38:04 43985:43985 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:38:04 43984:43984 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7727731Z 
2022-12-01T10:41:27.7728064Z STAGE:2022-12-01 10:38:04 43985:43985 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7728396Z STAGE:2022-12-01 10:38:04 43984:43984 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7728730Z STAGE:2022-12-01 10:38:04 43985:43985 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7729057Z STAGE:2022-12-01 10:38:04 43984:43984 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7729143Z ok (7.095s)
2022-12-01T10:41:27.7729165Z 
2022-12-01T10:41:27.7729430Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7729546Z Ran 1 test in 7.096s
2022-12-01T10:41:27.7729573Z 
2022-12-01T10:41:27.7729665Z OK
2022-12-01T10:41:27.7729686Z 
2022-12-01T10:41:27.7729858Z Generating XML reports...
2022-12-01T10:41:27.7730320Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103758.xml
2022-12-01T10:41:27.7730694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7730869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7731232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7731425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7731448Z 
2022-12-01T10:41:27.7731557Z Running tests...
2022-12-01T10:41:27.7731824Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7732135Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7732400Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7732625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44332
2022-12-01T10:41:27.7732843Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44333
2022-12-01T10:41:27.7733213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7733373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7733751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7733940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7734305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7734537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7734916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7735106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7735354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7735581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7736078Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7736475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7736704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7736937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7737096Z skip: Skipped due to small world size. (3.958s)
2022-12-01T10:41:27.7737117Z 
2022-12-01T10:41:27.7737384Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7737503Z Ran 1 test in 3.958s
2022-12-01T10:41:27.7737523Z 
2022-12-01T10:41:27.7737631Z OK (skipped=1)
2022-12-01T10:41:27.7737650Z 
2022-12-01T10:41:27.7737755Z Generating XML reports...
2022-12-01T10:41:27.7738204Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103808.xml
2022-12-01T10:41:27.7738574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7738749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7739133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7739370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7739392Z 
2022-12-01T10:41:27.7739508Z Running tests...
2022-12-01T10:41:27.7739773Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7740083Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7740319Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7740536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44640
2022-12-01T10:41:27.7740752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44641
2022-12-01T10:41:27.7741119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7741295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7741678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7741867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7742236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7742393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7742769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7742959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7743203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7743446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7743907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7744300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7744530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7744758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7744900Z skip: Skipped due to small world size. (4.089s)
2022-12-01T10:41:27.7744920Z 
2022-12-01T10:41:27.7745184Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7745351Z Ran 1 test in 4.089s
2022-12-01T10:41:27.7745371Z 
2022-12-01T10:41:27.7745479Z OK (skipped=1)
2022-12-01T10:41:27.7745498Z 
2022-12-01T10:41:27.7745624Z Generating XML reports...
2022-12-01T10:41:27.7746073Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103814.xml
2022-12-01T10:41:27.7746456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7746633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7747014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7747188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7747209Z 
2022-12-01T10:41:27.7747316Z Running tests...
2022-12-01T10:41:27.7747581Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7747893Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7748159Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7748381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44902
2022-12-01T10:41:27.7748642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44903
2022-12-01T10:41:27.7749025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7749183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7749739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7749937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7750313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7750487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7750862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7751060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7751304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7751538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7751945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7752347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7752579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7752808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7752967Z skip: Skipped due to small world size. (3.935s)
2022-12-01T10:41:27.7753058Z 
2022-12-01T10:41:27.7753338Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7753433Z Ran 1 test in 3.936s
2022-12-01T10:41:27.7753454Z 
2022-12-01T10:41:27.7753562Z OK (skipped=1)
2022-12-01T10:41:27.7753581Z 
2022-12-01T10:41:27.7753706Z Generating XML reports...
2022-12-01T10:41:27.7754155Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103820.xml
2022-12-01T10:41:27.7754527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7754701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7755080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7755272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7755296Z 
2022-12-01T10:41:27.7755405Z Running tests...
2022-12-01T10:41:27.7755652Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7755962Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7756217Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7756438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45190
2022-12-01T10:41:27.7756655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45191
2022-12-01T10:41:27.7757023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7757197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7757574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7757756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7758190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7758374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7758753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7758989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7759243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7759489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7759890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7760296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7760512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7760741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7760899Z skip: Skipped due to small world size. (4.109s)
2022-12-01T10:41:27.7760919Z 
2022-12-01T10:41:27.7761189Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7761303Z Ran 1 test in 4.109s
2022-12-01T10:41:27.7761324Z 
2022-12-01T10:41:27.7761430Z OK (skipped=1)
2022-12-01T10:41:27.7761449Z 
2022-12-01T10:41:27.7761573Z Generating XML reports...
2022-12-01T10:41:27.7762022Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103826.xml
2022-12-01T10:41:27.7762375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7762611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7762991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7763185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7763205Z 
2022-12-01T10:41:27.7763314Z Running tests...
2022-12-01T10:41:27.7763580Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7763893Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7764141Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7764361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45485
2022-12-01T10:41:27.7764563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45486
2022-12-01T10:41:27.7764943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7765119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7765500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7765689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7766054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7766230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7766608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7766782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7767033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7767320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7767732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7768129Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7768359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7768691Z STAGE:2022-12-01 10:38:37 45485:45485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7768919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7769250Z STAGE:2022-12-01 10:38:37 45486:45486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7769480Z [1669891118.936461] [7ff5451e42d4:45486:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7769724Z [1669891118.936461] [7ff5451e42d4:45486:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7769952Z [1669891118.912486] [7ff5451e42d4:45485:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7770181Z [1669891118.912486] [7ff5451e42d4:45485:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7770731Z STAGE:2022-12-01 10:38:39 45486:45486 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:38:39 45485:45485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7770753Z 
2022-12-01T10:41:27.7771082Z STAGE:2022-12-01 10:38:39 45486:45486 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7771466Z STAGE:2022-12-01 10:38:39 45485:45485 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7771798Z STAGE:2022-12-01 10:38:39 45486:45486 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7772124Z STAGE:2022-12-01 10:38:39 45485:45485 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7772229Z ok (7.276s)
2022-12-01T10:41:27.7772249Z 
2022-12-01T10:41:27.7772498Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7772612Z Ran 1 test in 7.277s
2022-12-01T10:41:27.7772632Z 
2022-12-01T10:41:27.7772724Z OK
2022-12-01T10:41:27.7772743Z 
2022-12-01T10:41:27.7772868Z Generating XML reports...
2022-12-01T10:41:27.7773319Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103833.xml
2022-12-01T10:41:27.7773689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7773874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7774253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7774446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7774466Z 
2022-12-01T10:41:27.7774557Z Running tests...
2022-12-01T10:41:27.7774823Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7775136Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7775381Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7775600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45807
2022-12-01T10:41:27.7775818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45808
2022-12-01T10:41:27.7776245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7776428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7776820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7777013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7777377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7777553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7777929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7778118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7778364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7778613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7779013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7779396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7779625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7779960Z STAGE:2022-12-01 10:38:46 45807:45807 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7780186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7780517Z STAGE:2022-12-01 10:38:46 45808:45808 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7780825Z [1669891128.254385] [7ff5451e42d4:45808:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7781113Z [1669891128.254385] [7ff5451e42d4:45808:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7781338Z [1669891128.243296] [7ff5451e42d4:45807:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7781566Z [1669891128.243296] [7ff5451e42d4:45807:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7782104Z STAGE:2022-12-01 10:38:48 45808:45808 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:38:48 45807:45807 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7782144Z 
2022-12-01T10:41:27.7782453Z STAGE:2022-12-01 10:38:48 45807:45807 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7782797Z STAGE:2022-12-01 10:38:48 45807:45807 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7783118Z STAGE:2022-12-01 10:38:48 45808:45808 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7783445Z STAGE:2022-12-01 10:38:48 45808:45808 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7783550Z ok (7.064s)
2022-12-01T10:41:27.7783571Z 
2022-12-01T10:41:27.7783836Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7783952Z Ran 1 test in 7.064s
2022-12-01T10:41:27.7783972Z 
2022-12-01T10:41:27.7784064Z OK
2022-12-01T10:41:27.7784083Z 
2022-12-01T10:41:27.7784206Z Generating XML reports...
2022-12-01T10:41:27.7784637Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103842.xml
2022-12-01T10:41:27.7785011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7785250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7785645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7785837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7785858Z 
2022-12-01T10:41:27.7785964Z Running tests...
2022-12-01T10:41:27.7786229Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7786538Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7786797Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-12-01T10:41:27.7786836Z 
2022-12-01T10:41:27.7787079Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7787192Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7787216Z 
2022-12-01T10:41:27.7787326Z OK (skipped=1)
2022-12-01T10:41:27.7787346Z 
2022-12-01T10:41:27.7787472Z Generating XML reports...
2022-12-01T10:41:27.7787921Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103852.xml
2022-12-01T10:41:27.7788291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7788468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7788851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7789026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7789066Z 
2022-12-01T10:41:27.7789156Z Running tests...
2022-12-01T10:41:27.7789606Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7789934Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7790285Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7790507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46222
2022-12-01T10:41:27.7790721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46223
2022-12-01T10:41:27.7791095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7791252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7791635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7791829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7792199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7792381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7792760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7792952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7793197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7793443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7793826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7794223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7794456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7794852Z STAGE:2022-12-01 10:38:57 46222:46222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7795092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7795427Z STAGE:2022-12-01 10:38:57 46223:46223 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7795665Z [1669891139.232107] [7ff5451e42d4:46222:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7795905Z [1669891139.232107] [7ff5451e42d4:46222:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7796127Z [1669891139.254163] [7ff5451e42d4:46223:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7796337Z [1669891139.254163] [7ff5451e42d4:46223:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7796898Z STAGE:2022-12-01 10:38:59 46222:46222 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:38:59 46223:46223 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7796939Z 
2022-12-01T10:41:27.7797250Z STAGE:2022-12-01 10:38:59 46223:46223 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7797571Z STAGE:2022-12-01 10:38:59 46222:46222 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7797902Z STAGE:2022-12-01 10:38:59 46223:46223 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7798228Z STAGE:2022-12-01 10:38:59 46222:46222 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7798330Z ok (6.945s)
2022-12-01T10:41:27.7798350Z 
2022-12-01T10:41:27.7798617Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7798732Z Ran 1 test in 6.946s
2022-12-01T10:41:27.7798799Z 
2022-12-01T10:41:27.7798895Z OK
2022-12-01T10:41:27.7798914Z 
2022-12-01T10:41:27.7799024Z Generating XML reports...
2022-12-01T10:41:27.7799476Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103854.xml
2022-12-01T10:41:27.7799849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7800026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7800409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7800601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7800621Z 
2022-12-01T10:41:27.7800730Z Running tests...
2022-12-01T10:41:27.7800995Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7801286Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7801582Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-12-01T10:41:27.7801603Z 
2022-12-01T10:41:27.7801866Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7801978Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7801999Z 
2022-12-01T10:41:27.7802107Z OK (skipped=1)
2022-12-01T10:41:27.7802126Z 
2022-12-01T10:41:27.7802246Z Generating XML reports...
2022-12-01T10:41:27.7802695Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103903.xml
2022-12-01T10:41:27.7803063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7803239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7803601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7803843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7803864Z 
2022-12-01T10:41:27.7803977Z Running tests...
2022-12-01T10:41:27.7804243Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7804555Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7804830Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-12-01T10:41:27.7804851Z 
2022-12-01T10:41:27.7805113Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7805224Z Ran 1 test in 0.003s
2022-12-01T10:41:27.7805244Z 
2022-12-01T10:41:27.7805351Z OK (skipped=1)
2022-12-01T10:41:27.7805370Z 
2022-12-01T10:41:27.7805476Z Generating XML reports...
2022-12-01T10:41:27.7805925Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103905.xml
2022-12-01T10:41:27.7806303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7806481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7806862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7807051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7807072Z 
2022-12-01T10:41:27.7807181Z Running tests...
2022-12-01T10:41:27.7807443Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7807753Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7807982Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7808202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46404
2022-12-01T10:41:27.7808480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46405
2022-12-01T10:41:27.7808853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7809029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7809408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7809599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7809969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7810126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7810501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7810699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7810946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7811191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7811594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7811995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7812226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7812562Z STAGE:2022-12-01 10:39:10 46404:46404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7812774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7813151Z STAGE:2022-12-01 10:39:10 46405:46405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7813396Z [1669891152.650027] [7ff5451e42d4:46404:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7813638Z [1669891152.650027] [7ff5451e42d4:46404:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7813863Z [1669891152.607959] [7ff5451e42d4:46405:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7814093Z [1669891152.607959] [7ff5451e42d4:46405:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7814644Z STAGE:2022-12-01 10:39:13 46404:46404 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:39:13 46405:46405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7814669Z 
2022-12-01T10:41:27.7815003Z STAGE:2022-12-01 10:39:13 46405:46405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7815325Z STAGE:2022-12-01 10:39:13 46404:46404 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7815663Z STAGE:2022-12-01 10:39:13 46405:46405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7815970Z STAGE:2022-12-01 10:39:13 46404:46404 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7816076Z ok (7.093s)
2022-12-01T10:41:27.7816096Z 
2022-12-01T10:41:27.7816362Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7816474Z Ran 1 test in 7.094s
2022-12-01T10:41:27.7816494Z 
2022-12-01T10:41:27.7816586Z OK
2022-12-01T10:41:27.7816606Z 
2022-12-01T10:41:27.7816729Z Generating XML reports...
2022-12-01T10:41:27.7817176Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103907.xml
2022-12-01T10:41:27.7817610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7817787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7818151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7818344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7818365Z 
2022-12-01T10:41:27.7818471Z Running tests...
2022-12-01T10:41:27.7818734Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7819047Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7819306Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:41:27.7819330Z 
2022-12-01T10:41:27.7819590Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7819707Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7819727Z 
2022-12-01T10:41:27.7819817Z OK (skipped=1)
2022-12-01T10:41:27.7819855Z 
2022-12-01T10:41:27.7819961Z Generating XML reports...
2022-12-01T10:41:27.7820406Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103916.xml
2022-12-01T10:41:27.7820777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7820953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7821334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7821524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7821544Z 
2022-12-01T10:41:27.7821652Z Running tests...
2022-12-01T10:41:27.7821921Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7822261Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7822537Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:41:27.7822558Z 
2022-12-01T10:41:27.7822817Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7822931Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7822950Z 
2022-12-01T10:41:27.7823062Z OK (skipped=1)
2022-12-01T10:41:27.7823081Z 
2022-12-01T10:41:27.7823204Z Generating XML reports...
2022-12-01T10:41:27.7823649Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103918.xml
2022-12-01T10:41:27.7824021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7824201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7824571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7824764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7824786Z 
2022-12-01T10:41:27.7824893Z Running tests...
2022-12-01T10:41:27.7825153Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7825465Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7825726Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7825947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46927
2022-12-01T10:41:27.7826166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46928
2022-12-01T10:41:27.7826519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7826755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7827135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7827324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7827690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7827866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7828246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7828437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7828681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7828914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7829318Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7829910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7830144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7830481Z STAGE:2022-12-01 10:39:24 46928:46928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7830707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7831037Z STAGE:2022-12-01 10:39:24 46927:46927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7831273Z [1669891165.940745] [7ff5451e42d4:46927:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7831586Z [1669891165.940745] [7ff5451e42d4:46927:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7831807Z [1669891165.981987] [7ff5451e42d4:46928:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7832035Z [1669891165.981987] [7ff5451e42d4:46928:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7832589Z STAGE:2022-12-01 10:39:26 46927:46927 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:39:26 46928:46928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7832611Z 
2022-12-01T10:41:27.7832939Z STAGE:2022-12-01 10:39:26 46927:46927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7833260Z STAGE:2022-12-01 10:39:26 46928:46928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7833598Z STAGE:2022-12-01 10:39:26 46927:46927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7833926Z STAGE:2022-12-01 10:39:26 46928:46928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7834029Z ok (7.026s)
2022-12-01T10:41:27.7834050Z 
2022-12-01T10:41:27.7834315Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7834428Z Ran 1 test in 7.026s
2022-12-01T10:41:27.7834448Z 
2022-12-01T10:41:27.7834523Z OK
2022-12-01T10:41:27.7834542Z 
2022-12-01T10:41:27.7834666Z Generating XML reports...
2022-12-01T10:41:27.7835116Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103920.xml
2022-12-01T10:41:27.7835490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7835735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7836119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7836311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7836332Z 
2022-12-01T10:41:27.7836440Z Running tests...
2022-12-01T10:41:27.7836688Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7836999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7837259Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7837280Z 
2022-12-01T10:41:27.7837537Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7837648Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7837668Z 
2022-12-01T10:41:27.7837774Z OK (skipped=1)
2022-12-01T10:41:27.7837797Z 
2022-12-01T10:41:27.7837920Z Generating XML reports...
2022-12-01T10:41:27.7838367Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103929.xml
2022-12-01T10:41:27.7838738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7838896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7839273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7839461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7839481Z 
2022-12-01T10:41:27.7839589Z Running tests...
2022-12-01T10:41:27.7839848Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7840155Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7840471Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7840493Z 
2022-12-01T10:41:27.7840762Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7840875Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7840895Z 
2022-12-01T10:41:27.7840984Z OK (skipped=1)
2022-12-01T10:41:27.7841003Z 
2022-12-01T10:41:27.7841124Z Generating XML reports...
2022-12-01T10:41:27.7841572Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103931.xml
2022-12-01T10:41:27.7841944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7842121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7842499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7842694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7842717Z 
2022-12-01T10:41:27.7842828Z Running tests...
2022-12-01T10:41:27.7843075Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7843390Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7843661Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7843682Z 
2022-12-01T10:41:27.7843945Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7844060Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7844079Z 
2022-12-01T10:41:27.7844190Z OK (skipped=1)
2022-12-01T10:41:27.7844209Z 
2022-12-01T10:41:27.7844334Z Generating XML reports...
2022-12-01T10:41:27.7844779Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103933.xml
2022-12-01T10:41:27.7845224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7845382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7845763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7845954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7845974Z 
2022-12-01T10:41:27.7846081Z Running tests...
2022-12-01T10:41:27.7846341Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7846650Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7846905Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:41:27.7846926Z 
2022-12-01T10:41:27.7847186Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7847304Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7847323Z 
2022-12-01T10:41:27.7847415Z OK (skipped=1)
2022-12-01T10:41:27.7847435Z 
2022-12-01T10:41:27.7847562Z Generating XML reports...
2022-12-01T10:41:27.7848003Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103936.xml
2022-12-01T10:41:27.7848373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7848547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7848927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7849117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7849137Z 
2022-12-01T10:41:27.7849245Z Running tests...
2022-12-01T10:41:27.7849490Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7849850Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7850126Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:41:27.7850147Z 
2022-12-01T10:41:27.7850405Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7850516Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7850536Z 
2022-12-01T10:41:27.7850645Z OK (skipped=1)
2022-12-01T10:41:27.7850664Z 
2022-12-01T10:41:27.7850786Z Generating XML reports...
2022-12-01T10:41:27.7851224Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103938.xml
2022-12-01T10:41:27.7851593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7851749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7852138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7852329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7852348Z 
2022-12-01T10:41:27.7852455Z Running tests...
2022-12-01T10:41:27.7852718Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7853027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7853299Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7853320Z 
2022-12-01T10:41:27.7853577Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7853691Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7853711Z 
2022-12-01T10:41:27.7853801Z OK (skipped=1)
2022-12-01T10:41:27.7853820Z 
2022-12-01T10:41:27.7853944Z Generating XML reports...
2022-12-01T10:41:27.7854476Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103940.xml
2022-12-01T10:41:27.7854846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7855025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7855403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7855595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7855616Z 
2022-12-01T10:41:27.7855722Z Running tests...
2022-12-01T10:41:27.7855983Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7856276Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7856548Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T10:41:27.7856572Z 
2022-12-01T10:41:27.7856838Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7856948Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7856968Z 
2022-12-01T10:41:27.7857076Z OK (skipped=1)
2022-12-01T10:41:27.7857097Z 
2022-12-01T10:41:27.7857218Z Generating XML reports...
2022-12-01T10:41:27.7857663Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103942.xml
2022-12-01T10:41:27.7858032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7858189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7858570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7858763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7858786Z 
2022-12-01T10:41:27.7858897Z Running tests...
2022-12-01T10:41:27.7859198Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7859518Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7859903Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:41:27.7859922Z 
2022-12-01T10:41:27.7860179Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7860289Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7860309Z 
2022-12-01T10:41:27.7860398Z OK (skipped=1)
2022-12-01T10:41:27.7860434Z 
2022-12-01T10:41:27.7860540Z Generating XML reports...
2022-12-01T10:41:27.7860977Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103944.xml
2022-12-01T10:41:27.7861344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7861527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7861903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7862092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7862113Z 
2022-12-01T10:41:27.7862221Z Running tests...
2022-12-01T10:41:27.7862481Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7862773Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7863016Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7863237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47849
2022-12-01T10:41:27.7863455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47850
2022-12-01T10:41:27.7863888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7864060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7864438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7864628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7864974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7865151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7865522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7865710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7865962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7866208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7866606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7867003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7867232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7867442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7867677Z [1669891191.492742] [7ff5451e42d4:47850:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7867918Z [1669891191.492742] [7ff5451e42d4:47850:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7868189Z [1669891191.498873] [7ff5451e42d4:47849:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7868426Z [1669891191.498873] [7ff5451e42d4:47849:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7868530Z ok (5.930s)
2022-12-01T10:41:27.7868550Z 
2022-12-01T10:41:27.7868819Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7868932Z Ran 1 test in 5.930s
2022-12-01T10:41:27.7868953Z 
2022-12-01T10:41:27.7869046Z OK
2022-12-01T10:41:27.7869066Z 
2022-12-01T10:41:27.7869170Z Generating XML reports...
2022-12-01T10:41:27.7869864Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103946.xml
2022-12-01T10:41:27.7870254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7870438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7870817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7871011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7871032Z 
2022-12-01T10:41:27.7871142Z Running tests...
2022-12-01T10:41:27.7871411Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7871724Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7871991Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-12-01T10:41:27.7872011Z 
2022-12-01T10:41:27.7872273Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7872475Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7872496Z 
2022-12-01T10:41:27.7872604Z OK (skipped=1)
2022-12-01T10:41:27.7872627Z 
2022-12-01T10:41:27.7872750Z Generating XML reports...
2022-12-01T10:41:27.7873198Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103954.xml
2022-12-01T10:41:27.7873565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7873740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7874117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7874291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7874311Z 
2022-12-01T10:41:27.7874422Z Running tests...
2022-12-01T10:41:27.7874684Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7874998Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7875309Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-12-01T10:41:27.7875331Z 
2022-12-01T10:41:27.7875590Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7875703Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7875722Z 
2022-12-01T10:41:27.7875827Z OK (skipped=1)
2022-12-01T10:41:27.7875847Z 
2022-12-01T10:41:27.7875953Z Generating XML reports...
2022-12-01T10:41:27.7876402Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103956.xml
2022-12-01T10:41:27.7876772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7876947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7877386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7877587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7877607Z 
2022-12-01T10:41:27.7877715Z Running tests...
2022-12-01T10:41:27.7877979Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7878294Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7878578Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-12-01T10:41:27.7878615Z 
2022-12-01T10:41:27.7878860Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7878970Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7878990Z 
2022-12-01T10:41:27.7879096Z OK (skipped=1)
2022-12-01T10:41:27.7879116Z 
2022-12-01T10:41:27.7879239Z Generating XML reports...
2022-12-01T10:41:27.7879688Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103958.xml
2022-12-01T10:41:27.7880059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7880236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7880613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7880787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7880808Z 
2022-12-01T10:41:27.7880918Z Running tests...
2022-12-01T10:41:27.7881225Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7881536Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7881811Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7882092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48543
2022-12-01T10:41:27.7882310Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48544
2022-12-01T10:41:27.7882687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7882844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7883225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7883421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7883785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7883961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7884346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7884539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7884789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7885034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7885415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7885812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7886043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7886380Z STAGE:2022-12-01 10:40:04 48544:48544 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7886659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7887004Z STAGE:2022-12-01 10:40:04 48543:48543 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7887245Z [1669891206.294799] [7ff5451e42d4:48544:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7887485Z [1669891206.294799] [7ff5451e42d4:48544:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7887822Z STAGE:2022-12-01 10:40:06 48544:48544 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7888037Z [1669891206.313992] [7ff5451e42d4:48543:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7888273Z [1669891206.313992] [7ff5451e42d4:48543:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7888617Z STAGE:2022-12-01 10:40:06 48543:48543 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7888720Z ok (7.047s)
2022-12-01T10:41:27.7888740Z 
2022-12-01T10:41:27.7889006Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7889120Z Ran 1 test in 7.047s
2022-12-01T10:41:27.7889140Z 
2022-12-01T10:41:27.7889232Z OK
2022-12-01T10:41:27.7889251Z 
2022-12-01T10:41:27.7889377Z Generating XML reports...
2022-12-01T10:41:27.7889822Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104000.xml
2022-12-01T10:41:27.7890173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7890354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7890732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7890981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7891001Z 
2022-12-01T10:41:27.7891109Z Running tests...
2022-12-01T10:41:27.7891376Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7891690Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7891925Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:41:27.7891946Z 
2022-12-01T10:41:27.7902673Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7902834Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7902856Z 
2022-12-01T10:41:27.7902970Z OK (skipped=1)
2022-12-01T10:41:27.7902991Z 
2022-12-01T10:41:27.7903121Z Generating XML reports...
2022-12-01T10:41:27.7903601Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104010.xml
2022-12-01T10:41:27.7903998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7904174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7904559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7904752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7904772Z 
2022-12-01T10:41:27.7904883Z Running tests...
2022-12-01T10:41:27.7905151Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7905464Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7905719Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:41:27.7905758Z 
2022-12-01T10:41:27.7906010Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7906236Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7906260Z 
2022-12-01T10:41:27.7906378Z OK (skipped=1)
2022-12-01T10:41:27.7906398Z 
2022-12-01T10:41:27.7906522Z Generating XML reports...
2022-12-01T10:41:27.7906972Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104012.xml
2022-12-01T10:41:27.7907345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7907522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7907905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7908079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7908117Z 
2022-12-01T10:41:27.7908210Z Running tests...
2022-12-01T10:41:27.7908479Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7908795Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7909057Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:41:27.7909077Z 
2022-12-01T10:41:27.7909341Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7909851Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7909873Z 
2022-12-01T10:41:27.7909981Z OK (skipped=1)
2022-12-01T10:41:27.7910001Z 
2022-12-01T10:41:27.7910125Z Generating XML reports...
2022-12-01T10:41:27.7910563Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104014.xml
2022-12-01T10:41:27.7910934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7911109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7911609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7911877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7911901Z 
2022-12-01T10:41:27.7912010Z Running tests...
2022-12-01T10:41:27.7912272Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7912586Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7912839Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7913059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49207
2022-12-01T10:41:27.7913279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49208
2022-12-01T10:41:27.7913652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7913838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7914220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7914409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7914776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7914949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7915308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7915499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7915743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7916077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7916500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7916897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7917126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7917462Z STAGE:2022-12-01 10:40:19 49208:49208 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7917692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7918010Z STAGE:2022-12-01 10:40:19 49207:49207 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7918249Z [1669891221.724508] [7ff5451e42d4:49208:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7918500Z [1669891221.724508] [7ff5451e42d4:49208:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7918842Z STAGE:2022-12-01 10:40:22 49208:49208 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7919072Z [1669891221.662417] [7ff5451e42d4:49207:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7919299Z [1669891221.662417] [7ff5451e42d4:49207:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7919629Z STAGE:2022-12-01 10:40:22 49207:49207 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7919731Z ok (7.080s)
2022-12-01T10:41:27.7919752Z 
2022-12-01T10:41:27.7920016Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7920113Z Ran 1 test in 7.080s
2022-12-01T10:41:27.7920207Z 
2022-12-01T10:41:27.7920306Z OK
2022-12-01T10:41:27.7920326Z 
2022-12-01T10:41:27.7920457Z Generating XML reports...
2022-12-01T10:41:27.7920908Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104016.xml
2022-12-01T10:41:27.7921278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7921453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7921836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7922032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7922052Z 
2022-12-01T10:41:27.7922160Z Running tests...
2022-12-01T10:41:27.7922406Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7922714Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7922979Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7923197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49645
2022-12-01T10:41:27.7923412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49646
2022-12-01T10:41:27.7923782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7923958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7924334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7924509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7924879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7925104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7925495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7925686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7925929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7926173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7926572Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7926970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7927183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7927416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7927651Z [1669891230.755957] [7ff5451e42d4:49645:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7927889Z [1669891230.755957] [7ff5451e42d4:49645:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7928120Z [1669891230.784486] [7ff5451e42d4:49646:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7928347Z [1669891230.784486] [7ff5451e42d4:49646:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7928449Z ok (6.007s)
2022-12-01T10:41:27.7928469Z 
2022-12-01T10:41:27.7928741Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7928853Z Ran 1 test in 6.008s
2022-12-01T10:41:27.7928921Z 
2022-12-01T10:41:27.7929001Z OK
2022-12-01T10:41:27.7929020Z 
2022-12-01T10:41:27.7929151Z Generating XML reports...
2022-12-01T10:41:27.7929603Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104025.xml
2022-12-01T10:41:27.7929976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7930153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7930530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7930721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7930741Z 
2022-12-01T10:41:27.7930848Z Running tests...
2022-12-01T10:41:27.7931096Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7931411Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7931705Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7931926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50034
2022-12-01T10:41:27.7932144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50035
2022-12-01T10:41:27.7932518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7932694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7933071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7933261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7933613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7933839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7934231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7934420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7934662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7934904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7935302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7935696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7935921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7936249Z STAGE:2022-12-01 10:40:37 50034:50034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7936478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7936809Z STAGE:2022-12-01 10:40:37 50035:50035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7937041Z [1669891239.480331] [7ff5451e42d4:50034:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7937280Z [1669891239.480331] [7ff5451e42d4:50034:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7937615Z STAGE:2022-12-01 10:40:39 50034:50034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7937841Z [1669891239.520554] [7ff5451e42d4:50035:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7938130Z [1669891239.520554] [7ff5451e42d4:50035:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7938465Z STAGE:2022-12-01 10:40:39 50035:50035 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7938550Z ok (7.172s)
2022-12-01T10:41:27.7938570Z 
2022-12-01T10:41:27.7938903Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7939018Z Ran 1 test in 7.172s
2022-12-01T10:41:27.7939038Z 
2022-12-01T10:41:27.7939132Z OK
2022-12-01T10:41:27.7939151Z 
2022-12-01T10:41:27.7939273Z Generating XML reports...
2022-12-01T10:41:27.7939723Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104034.xml
2022-12-01T10:41:27.7940093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7940267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7940636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7940829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7940850Z 
2022-12-01T10:41:27.7940958Z Running tests...
2022-12-01T10:41:27.7941221Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7941534Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7941813Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7942032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50471
2022-12-01T10:41:27.7942248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50472
2022-12-01T10:41:27.7942616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7942827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7943214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7943405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7943771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7943945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7944320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7944509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7944754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7944988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7945392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7945793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7946022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7946356Z STAGE:2022-12-01 10:40:46 50471:50471 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7946583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7946912Z STAGE:2022-12-01 10:40:46 50472:50472 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:41:27.7947149Z [1669891248.705270] [7ff5451e42d4:50472:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7947447Z [1669891248.705270] [7ff5451e42d4:50472:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7947783Z STAGE:2022-12-01 10:40:49 50472:50472 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7947994Z [1669891248.676935] [7ff5451e42d4:50471:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7948224Z [1669891248.676935] [7ff5451e42d4:50471:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7948556Z STAGE:2022-12-01 10:40:49 50471:50471 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:41:27.7948658Z ok (7.086s)
2022-12-01T10:41:27.7948679Z 
2022-12-01T10:41:27.7948943Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7949056Z Ran 1 test in 7.086s
2022-12-01T10:41:27.7949079Z 
2022-12-01T10:41:27.7949171Z OK
2022-12-01T10:41:27.7949190Z 
2022-12-01T10:41:27.7949316Z Generating XML reports...
2022-12-01T10:41:27.7950007Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104043.xml
2022-12-01T10:41:27.7950382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7950558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7950941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7951133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7951153Z 
2022-12-01T10:41:27.7951259Z Running tests...
2022-12-01T10:41:27.7951521Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7951836Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7952194Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T10:41:27.7952216Z 
2022-12-01T10:41:27.7952471Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7952584Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7952603Z 
2022-12-01T10:41:27.7952712Z OK (skipped=1)
2022-12-01T10:41:27.7952732Z 
2022-12-01T10:41:27.7952855Z Generating XML reports...
2022-12-01T10:41:27.7953297Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104052.xml
2022-12-01T10:41:27.7953667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7953843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7954223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7954423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7954443Z 
2022-12-01T10:41:27.7954534Z Running tests...
2022-12-01T10:41:27.7954798Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7955111Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7955404Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T10:41:27.7955424Z 
2022-12-01T10:41:27.7955687Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7955798Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7955818Z 
2022-12-01T10:41:27.7955914Z OK (skipped=1)
2022-12-01T10:41:27.7955934Z 
2022-12-01T10:41:27.7956044Z Generating XML reports...
2022-12-01T10:41:27.7956477Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104054.xml
2022-12-01T10:41:27.7956910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7957076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7957454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7957646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7957666Z 
2022-12-01T10:41:27.7957772Z Running tests...
2022-12-01T10:41:27.7958036Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7958346Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7958613Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7958826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50971
2022-12-01T10:41:27.7959045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50972
2022-12-01T10:41:27.7959417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7959594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7959970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7960158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7960525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7960699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7961079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7961303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7961555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7961797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7962200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7962597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7962824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7963050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7963287Z [1669891262.203295] [7ff5451e42d4:50972:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7963530Z [1669891262.203295] [7ff5451e42d4:50972:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7963738Z [1669891262.198550] [7ff5451e42d4:50971:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7963966Z [1669891262.198550] [7ff5451e42d4:50971:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7964068Z ok (6.692s)
2022-12-01T10:41:27.7964088Z 
2022-12-01T10:41:27.7964355Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7964468Z Ran 1 test in 6.692s
2022-12-01T10:41:27.7964488Z 
2022-12-01T10:41:27.7964579Z OK
2022-12-01T10:41:27.7964598Z 
2022-12-01T10:41:27.7964721Z Generating XML reports...
2022-12-01T10:41:27.7965168Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104056.xml
2022-12-01T10:41:27.7965599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7965757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7966136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7966327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7966347Z 
2022-12-01T10:41:27.7966455Z Running tests...
2022-12-01T10:41:27.7966719Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7967027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7967292Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7967515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51388
2022-12-01T10:41:27.7967718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51389
2022-12-01T10:41:27.7968097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7968270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7968646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7968834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7969198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7969371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7969748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7969983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7970219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7970462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7970868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7971265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7971494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7971750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxgqcal1c
2022-12-01T10:41:27.7972022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxgqcal1c/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7972257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7972495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt6f_4_gp
2022-12-01T10:41:27.7972760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt6f_4_gp/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7972994Z [1669891271.155494] [7ff5451e42d4:51388:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7973233Z [1669891271.155494] [7ff5451e42d4:51388:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7973455Z [1669891271.168060] [7ff5451e42d4:51389:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7973684Z [1669891271.168060] [7ff5451e42d4:51389:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7973840Z ok (6.178s)
2022-12-01T10:41:27.7973864Z 
2022-12-01T10:41:27.7974135Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7974248Z Ran 1 test in 6.178s
2022-12-01T10:41:27.7974268Z 
2022-12-01T10:41:27.7974360Z OK
2022-12-01T10:41:27.7974381Z 
2022-12-01T10:41:27.7974487Z Generating XML reports...
2022-12-01T10:41:27.7974933Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104105.xml
2022-12-01T10:41:27.7975307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7975482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7975860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7976052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7976075Z 
2022-12-01T10:41:27.7976186Z Running tests...
2022-12-01T10:41:27.7976448Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7976745Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7977044Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl & Gloo backend support DistributedDataParallel (0.002s)
2022-12-01T10:41:27.7977064Z 
2022-12-01T10:41:27.7977321Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7977432Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7977452Z 
2022-12-01T10:41:27.7977558Z OK (skipped=1)
2022-12-01T10:41:27.7977577Z 
2022-12-01T10:41:27.7977699Z Generating XML reports...
2022-12-01T10:41:27.7978144Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104114.xml
2022-12-01T10:41:27.7978568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7978754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7979119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7979311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7979331Z 
2022-12-01T10:41:27.7979437Z Running tests...
2022-12-01T10:41:27.7979698Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7980005Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7980297Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:41:27.7980517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51832
2022-12-01T10:41:27.7980739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51833
2022-12-01T10:41:27.7981166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7981329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7981709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7981898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7982263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7982438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7982813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7983076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7983325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:41:27.7983553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:41:27.7983957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7984359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:41:27.7984589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:41:27.7984815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:41:27.7985071Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9blh2by8
2022-12-01T10:41:27.7985345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9blh2by8/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7985600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgyw3clgd
2022-12-01T10:41:27.7985869Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgyw3clgd/_remote_module_non_scriptable.py
2022-12-01T10:41:27.7986087Z [1669891281.490390] [7ff5451e42d4:51832:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7986326Z [1669891281.490390] [7ff5451e42d4:51832:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7987148Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:41:27.7987399Z [1669891281.491318] [7ff5451e42d4:51833:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T10:41:27.7987637Z [1669891281.491318] [7ff5451e42d4:51833:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T10:41:27.7988409Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:41:27.7988516Z ok (6.450s)
2022-12-01T10:41:27.7988539Z 
2022-12-01T10:41:27.7988817Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7988933Z Ran 1 test in 6.450s
2022-12-01T10:41:27.7988955Z 
2022-12-01T10:41:27.7989048Z OK
2022-12-01T10:41:27.7989067Z 
2022-12-01T10:41:27.7989190Z Generating XML reports...
2022-12-01T10:41:27.7989840Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104116.xml
2022-12-01T10:41:27.7990226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7990385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7990766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7990955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7991053Z 
2022-12-01T10:41:27.7991173Z Running tests...
2022-12-01T10:41:27.7991441Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7991751Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7992181Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T10:41:27.7992202Z 
2022-12-01T10:41:27.7992462Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7992573Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7992594Z 
2022-12-01T10:41:27.7992684Z OK (skipped=1)
2022-12-01T10:41:27.7992703Z 
2022-12-01T10:41:27.7992827Z Generating XML reports...
2022-12-01T10:41:27.7993276Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104125.xml
2022-12-01T10:41:27.7993653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:41:27.7993828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:41:27.7994204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:41:27.7994395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:41:27.7994416Z 
2022-12-01T10:41:27.7994523Z Running tests...
2022-12-01T10:41:27.7994770Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7995081Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T10:41:27.7995515Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T10:41:27.7995537Z 
2022-12-01T10:41:27.7995800Z ----------------------------------------------------------------------
2022-12-01T10:41:27.7995970Z Ran 1 test in 0.002s
2022-12-01T10:41:27.7995991Z 
2022-12-01T10:41:27.7996104Z OK (skipped=1)
2022-12-01T10:41:27.7996123Z 
2022-12-01T10:41:27.7996248Z Generating XML reports...
2022-12-01T10:41:27.7996695Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104127.xml
2022-12-01T10:41:27.7996716Z 
2022-12-01T10:41:27.7997155Z ##[endgroup]
2022-12-01T10:41:27.7997626Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_go5u7ya9)
2022-12-01T10:41:27.7997646Z 
2022-12-01T10:45:06.5125989Z 
2022-12-01T10:45:06.5126487Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:45:06.5129400Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_a2b6lcu2)
2022-12-01T10:45:06.5133518Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:45:06.5177898Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T10:45:06.5214631Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5215280Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5215740Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5216143Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5216598Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5217089Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5217579Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5218061Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5218578Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5219414Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5220054Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5220577Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5221119Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5221646Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5222132Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5222607Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5223081Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5223515Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5223934Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5224703Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5225288Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5225790Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5226249Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5226643Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5227108Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5227541Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5227953Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5228383Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5228799Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5229179Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5230005Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5230411Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5230809Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5231200Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5231631Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5232053Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5232447Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5232876Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5233298Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5233711Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5234209Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5234682Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5235137Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5235577Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5236019Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5236452Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5236868Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5237312Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5237730Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5238178Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5238622Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5239043Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5239456Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5239870Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5240312Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5240732Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5241157Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5241564Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5241970Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5242366Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5242847Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5243258Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5243697Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5244083Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5244461Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5244883Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5245300Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5245680Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5246075Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5246462Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5246850Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5247266Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5247674Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5248111Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5248509Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5248869Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5249259Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5249657Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5250049Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5250461Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5250859Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5251234Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5251648Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5252152Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5252610Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5253038Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5253504Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5253968Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5254428Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5254856Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5255300Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5255747Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5256184Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5256670Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5257137Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5257616Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5258062Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5258525Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5258952Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5259336Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5259726Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5260097Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5260515Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5260907Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5261305Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5261701Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5262070Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5262484Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5262927Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5263327Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5263768Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5264199Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5264643Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5265053Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5265481Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5265905Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5266327Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5266746Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5267185Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5267621Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5268000Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5268392Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5268789Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5269164Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5270069Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5270766Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5271250Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5271757Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5272223Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5272653Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5273056Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5273505Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5273964Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5274446Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5274890Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5275322Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5275768Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5276178Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5276564Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5276969Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5277392Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5277828Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5278297Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5278767Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5279332Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5279749Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5280227Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5280746Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5281319Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5281929Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5282568Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5283457Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5284093Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5284702Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5285318Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5285936Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5286497Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5287009Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5287740Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5288154Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5288573Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5288991Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5289378Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5289811Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5290260Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5290712Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5291193Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5291618Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5292006Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5292433Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5292881Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5293316Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5293720Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5294142Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5294578Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5295001Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5295432Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5295852Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5296336Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5296747Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5297160Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5297589Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5298039Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5298477Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5298863Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5299253Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5299685Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5300107Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5300497Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5300853Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5301234Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5301623Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5301989Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5302366Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5302760Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5303132Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5303502Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5303865Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5304252Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5304636Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5305036Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5305421Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5305809Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5306203Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5306615Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5307019Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5307482Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5307940Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5308365Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5308776Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5309218Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5310099Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5310521Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5310946Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5311371Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5311770Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5312192Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5312594Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5313001Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5313459Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5313961Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5314439Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5314978Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5315448Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5315914Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5316357Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5316769Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5317207Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5317701Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5318142Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5318636Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5319161Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5319656Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5320066Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5320467Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5320877Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5321270Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5321668Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5322058Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5322456Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5322832Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5323209Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5323654Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5324036Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5324425Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5324828Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5325221Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5325608Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5325986Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5326358Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5326750Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5327124Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5327497Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5327860Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5328252Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5328647Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5329029Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5329418Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5329809Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5330168Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5330551Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5330983Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5331438Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5331859Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5332267Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5332748Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5333171Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5333598Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5334001Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5334403Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5334845Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5335271Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5335685Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5336082Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5336488Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5336883Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5337293Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5337749Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5338203Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5338957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5339406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5339996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5340478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5340717Z 
2022-12-01T10:45:06.5340809Z Running tests...
2022-12-01T10:45:06.5341227Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5341833Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5342441Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5342998Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1481
2022-12-01T10:45:06.5343454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1482
2022-12-01T10:45:06.5344074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5344513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5345093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5345569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5346161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5346596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5347177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5347649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5348118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5348772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5349318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5350293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5350915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5351397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5351925Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5352768Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5353422Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5354252Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5354921Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5355499Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5356307Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5357221Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5357887Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5358703Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5359375Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5360251Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5360930Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5361504Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:45:06.5362332Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5363222Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:45:06.5363724Z ok (7.209s)
2022-12-01T10:45:06.5363883Z 
2022-12-01T10:45:06.5364159Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5364478Z Ran 1 test in 7.209s
2022-12-01T10:45:06.5364645Z 
2022-12-01T10:45:06.5364741Z OK
2022-12-01T10:45:06.5364877Z 
2022-12-01T10:45:06.5365001Z Generating XML reports...
2022-12-01T10:45:06.5365616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102113.xml
2022-12-01T10:45:06.5366325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5366790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5367375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5367839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5368076Z 
2022-12-01T10:45:06.5368283Z Running tests...
2022-12-01T10:45:06.5368704Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5369241Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5369764Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-12-01T10:45:06.5370081Z 
2022-12-01T10:45:06.5370349Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5370679Z Ran 1 test in 0.004s
2022-12-01T10:45:06.5370844Z 
2022-12-01T10:45:06.5370958Z OK (skipped=1)
2022-12-01T10:45:06.5371096Z 
2022-12-01T10:45:06.5371222Z Generating XML reports...
2022-12-01T10:45:06.5371831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102122.xml
2022-12-01T10:45:06.5372559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5373007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5373598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5374082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5374318Z 
2022-12-01T10:45:06.5374428Z Running tests...
2022-12-01T10:45:06.5374819Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5375353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5375870Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5376352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1928
2022-12-01T10:45:06.5376809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1929
2022-12-01T10:45:06.5377478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5377951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5378515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5378996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5379637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5380072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5380656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5381130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5381591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5382089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5382758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5383461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5383992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5384452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5384799Z ok (3.966s)
2022-12-01T10:45:06.5384953Z 
2022-12-01T10:45:06.5385225Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5385541Z Ran 1 test in 3.966s
2022-12-01T10:45:06.5385706Z 
2022-12-01T10:45:06.5385799Z OK
2022-12-01T10:45:06.5385998Z 
2022-12-01T10:45:06.5386125Z Generating XML reports...
2022-12-01T10:45:06.5386745Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102125.xml
2022-12-01T10:45:06.5387454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5387910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5388487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5388949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5389183Z 
2022-12-01T10:45:06.5389293Z Running tests...
2022-12-01T10:45:06.5390040Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5390581Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5391110Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5392175Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.701s)
2022-12-01T10:45:06.5392723Z 
2022-12-01T10:45:06.5392992Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5393331Z Ran 1 test in 1.701s
2022-12-01T10:45:06.5393494Z 
2022-12-01T10:45:06.5393584Z OK (skipped=1)
2022-12-01T10:45:06.5393742Z 
2022-12-01T10:45:06.5393870Z Generating XML reports...
2022-12-01T10:45:06.5394482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml
2022-12-01T10:45:06.5395204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5395730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5396332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5396808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5397043Z 
2022-12-01T10:45:06.5397156Z Running tests...
2022-12-01T10:45:06.5397547Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5398079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5398637Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-12-01T10:45:06.5398971Z 
2022-12-01T10:45:06.5399218Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5399559Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5399724Z 
2022-12-01T10:45:06.5399833Z OK (skipped=1)
2022-12-01T10:45:06.5399993Z 
2022-12-01T10:45:06.5400120Z Generating XML reports...
2022-12-01T10:45:06.5400709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml
2022-12-01T10:45:06.5401430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5401892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5402450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5402932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5403171Z 
2022-12-01T10:45:06.5403282Z Running tests...
2022-12-01T10:45:06.5403689Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5404307Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5404886Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-12-01T10:45:06.5405234Z 
2022-12-01T10:45:06.5405501Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5405837Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5405983Z 
2022-12-01T10:45:06.5406093Z OK (skipped=1)
2022-12-01T10:45:06.5406252Z 
2022-12-01T10:45:06.5406377Z Generating XML reports...
2022-12-01T10:45:06.5406985Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102137.xml
2022-12-01T10:45:06.5407690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5408153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5408742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5409219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5409434Z 
2022-12-01T10:45:06.5409545Z Running tests...
2022-12-01T10:45:06.5409953Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5410493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5411028Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5411562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2599
2022-12-01T10:45:06.5412016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2600
2022-12-01T10:45:06.5412635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5413133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5413729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5414211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5414785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5415241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5415822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5416296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5416738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5417241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5417921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5418624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5419134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5419616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5420126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph24b09yr
2022-12-01T10:45:06.5420659Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph24b09yr/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5421198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80ollhm9
2022-12-01T10:45:06.5421805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80ollhm9/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5422719Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5423258Z   warnings.warn(
2022-12-01T10:45:06.5424013Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5424567Z   warnings.warn(
2022-12-01T10:45:06.5424947Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5425425Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5425917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5426411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5426895Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5427364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5427848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5428325Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5428782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5429261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5430064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5430554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5430969Z ok (8.149s)
2022-12-01T10:45:06.5431130Z 
2022-12-01T10:45:06.5431412Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5431745Z Ran 1 test in 8.149s
2022-12-01T10:45:06.5431912Z 
2022-12-01T10:45:06.5431987Z OK
2022-12-01T10:45:06.5432125Z 
2022-12-01T10:45:06.5432251Z Generating XML reports...
2022-12-01T10:45:06.5432868Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102139.xml
2022-12-01T10:45:06.5433596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5434037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5434622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5435171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5435411Z 
2022-12-01T10:45:06.5435502Z Running tests...
2022-12-01T10:45:06.5435913Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5436448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5437022Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5437548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3056
2022-12-01T10:45:06.5438000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3057
2022-12-01T10:45:06.5438622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5439061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5439725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5440184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5440763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5441224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5441817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5442290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5442732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5443243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5443907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5444620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5445133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5445611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5446112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzaga178
2022-12-01T10:45:06.5446660Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzaga178/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5447182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdew7p2w
2022-12-01T10:45:06.5447723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdew7p2w/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5448698Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5449273Z   warnings.warn(
2022-12-01T10:45:06.5450016Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5450575Z   warnings.warn(
2022-12-01T10:45:06.5450952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5451429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5451918Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5452405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5452770Z ok (7.128s)
2022-12-01T10:45:06.5452906Z 
2022-12-01T10:45:06.5453184Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5453519Z Ran 1 test in 7.129s
2022-12-01T10:45:06.5453686Z 
2022-12-01T10:45:06.5453783Z OK
2022-12-01T10:45:06.5453922Z 
2022-12-01T10:45:06.5454029Z Generating XML reports...
2022-12-01T10:45:06.5454649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102149.xml
2022-12-01T10:45:06.5455368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5455828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5456400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5456881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5457177Z 
2022-12-01T10:45:06.5457291Z Running tests...
2022-12-01T10:45:06.5457683Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5458220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5458801Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5459357Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3408
2022-12-01T10:45:06.5459789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3409
2022-12-01T10:45:06.5460402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5460860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5461451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5461918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5462509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5462964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5463524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5464005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5464475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5464982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5465628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5466378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5466912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5467389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5467881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppk1e6c_d
2022-12-01T10:45:06.5468429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppk1e6c_d/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5468965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7wh_c06
2022-12-01T10:45:06.5469875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7wh_c06/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5470813Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5471378Z   warnings.warn(
2022-12-01T10:45:06.5472137Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5472668Z   warnings.warn(
2022-12-01T10:45:06.5473045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5473539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5474024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5474492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5474951Z ok (6.876s)
2022-12-01T10:45:06.5475101Z 
2022-12-01T10:45:06.5475381Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5475697Z Ran 1 test in 6.876s
2022-12-01T10:45:06.5475858Z 
2022-12-01T10:45:06.5475954Z OK
2022-12-01T10:45:06.5476090Z 
2022-12-01T10:45:06.5476215Z Generating XML reports...
2022-12-01T10:45:06.5476826Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102159.xml
2022-12-01T10:45:06.5477531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5477989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5478568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5479030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5479319Z 
2022-12-01T10:45:06.5479428Z Running tests...
2022-12-01T10:45:06.5479845Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5480379Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5480957Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5481533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3765
2022-12-01T10:45:06.5481979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3766
2022-12-01T10:45:06.5482591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5483030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5483613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5484167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5484752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5485200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5485777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5486252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5486696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5487197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5487860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5488571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5489085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5489563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5490064Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphxdcmnf4
2022-12-01T10:45:06.5490594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphxdcmnf4/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5491133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3lkdo_s
2022-12-01T10:45:06.5491676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3lkdo_s/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5492591Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5493189Z   warnings.warn(
2022-12-01T10:45:06.5493939Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5494490Z   warnings.warn(
2022-12-01T10:45:06.5494868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5495347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5495706Z ok (7.186s)
2022-12-01T10:45:06.5495857Z 
2022-12-01T10:45:06.5496130Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5496443Z Ran 1 test in 7.187s
2022-12-01T10:45:06.5496607Z 
2022-12-01T10:45:06.5496707Z OK
2022-12-01T10:45:06.5496843Z 
2022-12-01T10:45:06.5496968Z Generating XML reports...
2022-12-01T10:45:06.5497583Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102208.xml
2022-12-01T10:45:06.5498295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5498753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5499334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5499792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5500029Z 
2022-12-01T10:45:06.5500141Z Running tests...
2022-12-01T10:45:06.5500553Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5501091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5501722Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5502298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4118
2022-12-01T10:45:06.5502748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4119
2022-12-01T10:45:06.5503343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5503803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5504385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5504863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5505432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5505894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5506476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5506952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5507391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5507890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5508551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5509235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5510086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5510666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5511170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0d6l3hg
2022-12-01T10:45:06.5511697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0d6l3hg/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5512232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqym9rm3g
2022-12-01T10:45:06.5512770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqym9rm3g/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5513694Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5514236Z   warnings.warn(
2022-12-01T10:45:06.5514996Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5515553Z   warnings.warn(
2022-12-01T10:45:06.5515937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5516411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5516896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5517381Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5517717Z ok (8.019s)
2022-12-01T10:45:06.5517867Z 
2022-12-01T10:45:06.5518140Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5518470Z Ran 1 test in 8.019s
2022-12-01T10:45:06.5518634Z 
2022-12-01T10:45:06.5518728Z OK
2022-12-01T10:45:06.5518850Z 
2022-12-01T10:45:06.5518975Z Generating XML reports...
2022-12-01T10:45:06.5519657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102217.xml
2022-12-01T10:45:06.5520396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5520833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5521416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5521893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5522130Z 
2022-12-01T10:45:06.5522243Z Running tests...
2022-12-01T10:45:06.5522633Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5523169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5523748Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5524276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4567
2022-12-01T10:45:06.5524734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4568
2022-12-01T10:45:06.5525340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5525798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5526360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5526838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5527426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5527921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5528506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5528976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5529437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5529920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5530581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5531279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5531816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5532281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5532785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9cii7wx
2022-12-01T10:45:06.5533333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9cii7wx/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5533852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo09vibe
2022-12-01T10:45:06.5534392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo09vibe/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5535307Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5535861Z   warnings.warn(
2022-12-01T10:45:06.5536650Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5537216Z   warnings.warn(
2022-12-01T10:45:06.5537593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5538084Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5538554Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5539041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5539397Z ok (7.442s)
2022-12-01T10:45:06.5539549Z 
2022-12-01T10:45:06.5539805Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5540139Z Ran 1 test in 7.442s
2022-12-01T10:45:06.5540302Z 
2022-12-01T10:45:06.5540397Z OK
2022-12-01T10:45:06.5540532Z 
2022-12-01T10:45:06.5540658Z Generating XML reports...
2022-12-01T10:45:06.5541262Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102227.xml
2022-12-01T10:45:06.5541992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5542451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5543014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5543490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5543737Z 
2022-12-01T10:45:06.5543852Z Running tests...
2022-12-01T10:45:06.5544242Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5544772Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5545361Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5545986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4922
2022-12-01T10:45:06.5546437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4923
2022-12-01T10:45:06.5547054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5547510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5548070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5548543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5549122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5549815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5550415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5550880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5551339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5551824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5552487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5553184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5553712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5554171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5554758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56hopsiw
2022-12-01T10:45:06.5555325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56hopsiw/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5555842Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfa4x8v13
2022-12-01T10:45:06.5556379Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfa4x8v13/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5557297Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5557854Z   warnings.warn(
2022-12-01T10:45:06.5558590Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.5559148Z   warnings.warn(
2022-12-01T10:45:06.5559527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5560017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5560485Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5560967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5561320Z ok (7.056s)
2022-12-01T10:45:06.5561468Z 
2022-12-01T10:45:06.5561722Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5562050Z Ran 1 test in 7.056s
2022-12-01T10:45:06.5562210Z 
2022-12-01T10:45:06.5562304Z OK
2022-12-01T10:45:06.5562436Z 
2022-12-01T10:45:06.5562560Z Generating XML reports...
2022-12-01T10:45:06.5563247Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102237.xml
2022-12-01T10:45:06.5563966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5564423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5564992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5565472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5565707Z 
2022-12-01T10:45:06.5565818Z Running tests...
2022-12-01T10:45:06.5566226Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5566742Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5567303Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5568383Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2022-12-01T10:45:06.5568913Z 
2022-12-01T10:45:06.5569183Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5569498Z Ran 1 test in 1.644s
2022-12-01T10:45:06.5569661Z 
2022-12-01T10:45:06.5569767Z OK (skipped=1)
2022-12-01T10:45:06.5569922Z 
2022-12-01T10:45:06.5570045Z Generating XML reports...
2022-12-01T10:45:06.5570635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102246.xml
2022-12-01T10:45:06.5571357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5571814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5572446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5572914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5573151Z 
2022-12-01T10:45:06.5573259Z Running tests...
2022-12-01T10:45:06.5573670Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5574199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5574737Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5575271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5394
2022-12-01T10:45:06.5575720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5395
2022-12-01T10:45:06.5576322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5576777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5577356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5577836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5578403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5578857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5579477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5579953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5580393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5580961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5581630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5582307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5582839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5583318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5583670Z ok (4.156s)
2022-12-01T10:45:06.5583803Z 
2022-12-01T10:45:06.5584073Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5584407Z Ran 1 test in 4.156s
2022-12-01T10:45:06.5584572Z 
2022-12-01T10:45:06.5584670Z OK
2022-12-01T10:45:06.5584806Z 
2022-12-01T10:45:06.5584912Z Generating XML reports...
2022-12-01T10:45:06.5585530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102250.xml
2022-12-01T10:45:06.5586253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5586711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5587273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5587746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5587980Z 
2022-12-01T10:45:06.5588088Z Running tests...
2022-12-01T10:45:06.5588477Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5589015Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5589920Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5591030Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2022-12-01T10:45:06.5591564Z 
2022-12-01T10:45:06.5591833Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5592144Z Ran 1 test in 1.667s
2022-12-01T10:45:06.5592307Z 
2022-12-01T10:45:06.5592418Z OK (skipped=1)
2022-12-01T10:45:06.5592572Z 
2022-12-01T10:45:06.5592698Z Generating XML reports...
2022-12-01T10:45:06.5593292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102257.xml
2022-12-01T10:45:06.5594027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5594508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5595110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5595594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5595826Z 
2022-12-01T10:45:06.5595935Z Running tests...
2022-12-01T10:45:06.5596341Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5596876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5597386Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5597889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5813
2022-12-01T10:45:06.5598436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5814
2022-12-01T10:45:06.5599035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5599485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5600054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5600521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5601077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5601529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5602099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5602559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5603000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5603495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5604151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5604826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5605351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5605822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5606175Z ok (6.819s)
2022-12-01T10:45:06.5606306Z 
2022-12-01T10:45:06.5606576Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5606908Z Ran 1 test in 6.819s
2022-12-01T10:45:06.5607071Z 
2022-12-01T10:45:06.5607165Z OK
2022-12-01T10:45:06.5607352Z 
2022-12-01T10:45:06.5607465Z Generating XML reports...
2022-12-01T10:45:06.5608078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102300.xml
2022-12-01T10:45:06.5608796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5609246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5609807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5610279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5610512Z 
2022-12-01T10:45:06.5610625Z Running tests...
2022-12-01T10:45:06.5611014Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5611553Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5612063Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T10:45:06.5612350Z 
2022-12-01T10:45:06.5612616Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5612927Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5613088Z 
2022-12-01T10:45:06.5613197Z OK (skipped=1)
2022-12-01T10:45:06.5613347Z 
2022-12-01T10:45:06.5613472Z Generating XML reports...
2022-12-01T10:45:06.5614057Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102309.xml
2022-12-01T10:45:06.5614771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5615221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5615795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5616317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5616550Z 
2022-12-01T10:45:06.5616658Z Running tests...
2022-12-01T10:45:06.5617068Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5617602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5618042Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5618518Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5618994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6285
2022-12-01T10:45:06.5619423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6286
2022-12-01T10:45:06.5620023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5620477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5621047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5621501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5622078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5622520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5623077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5623543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5623998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5624548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5625202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5625890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5626411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5626884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5627367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c_9qgyg
2022-12-01T10:45:06.5627904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c_9qgyg/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5628823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr33tduks
2022-12-01T10:45:06.5629599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr33tduks/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5629996Z ok (7.414s)
2022-12-01T10:45:06.5630146Z 
2022-12-01T10:45:06.5630434Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5630762Z Ran 1 test in 7.414s
2022-12-01T10:45:06.5630905Z 
2022-12-01T10:45:06.5631001Z OK
2022-12-01T10:45:06.5631137Z 
2022-12-01T10:45:06.5631264Z Generating XML reports...
2022-12-01T10:45:06.5631875Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102311.xml
2022-12-01T10:45:06.5632583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5633032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5633605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5634196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5634406Z 
2022-12-01T10:45:06.5634516Z Running tests...
2022-12-01T10:45:06.5634922Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5635452Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5635910Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5636418Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5636904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6726
2022-12-01T10:45:06.5637350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6727
2022-12-01T10:45:06.5637942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5638400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5638977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5639445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5640003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5640447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5641022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5641475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5641933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5642435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5643202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5643901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5644425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5644897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5645380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpox205vb9
2022-12-01T10:45:06.5645917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpox205vb9/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5646450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddrc18_4
2022-12-01T10:45:06.5646990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddrc18_4/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5647490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5647983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5648332Z ok (7.295s)
2022-12-01T10:45:06.5648479Z 
2022-12-01T10:45:06.5648752Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5649114Z Ran 1 test in 7.295s
2022-12-01T10:45:06.5649414Z 
2022-12-01T10:45:06.5649566Z OK
2022-12-01T10:45:06.5649817Z 
2022-12-01T10:45:06.5650008Z Generating XML reports...
2022-12-01T10:45:06.5650866Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102321.xml
2022-12-01T10:45:06.5651592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5652127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5652712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5653163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5653393Z 
2022-12-01T10:45:06.5653501Z Running tests...
2022-12-01T10:45:06.5653905Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5654419Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5654913Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5655447Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5655951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7143
2022-12-01T10:45:06.5656384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7144
2022-12-01T10:45:06.5656986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5657432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5657988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5658455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5659034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5659474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5660030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5660497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5661005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5661509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5662155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5662843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5663366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5663821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5664319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hv15q29
2022-12-01T10:45:06.5664853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hv15q29/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5665385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpixkv35np
2022-12-01T10:45:06.5665903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpixkv35np/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5666420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5666901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.5667236Z ok (7.332s)
2022-12-01T10:45:06.5667382Z 
2022-12-01T10:45:06.5667655Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5667984Z Ran 1 test in 7.332s
2022-12-01T10:45:06.5668145Z 
2022-12-01T10:45:06.5668240Z OK
2022-12-01T10:45:06.5668355Z 
2022-12-01T10:45:06.5668482Z Generating XML reports...
2022-12-01T10:45:06.5669091Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102331.xml
2022-12-01T10:45:06.5670235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5670671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5671246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5671714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5671946Z 
2022-12-01T10:45:06.5672057Z Running tests...
2022-12-01T10:45:06.5672444Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5672975Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5673446Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:45:06.5673925Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5674411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7553
2022-12-01T10:45:06.5674851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7554
2022-12-01T10:45:06.5675455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5675885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5676464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5676929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5677508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5677932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5678615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5679095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5679577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5680072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5680735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5681428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5681930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5682402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5682909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjynf_fxn
2022-12-01T10:45:06.5720317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjynf_fxn/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5721178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplp_tgw4a
2022-12-01T10:45:06.5721773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplp_tgw4a/_remote_module_non_scriptable.py
2022-12-01T10:45:06.5722171Z ok (7.110s)
2022-12-01T10:45:06.5722329Z 
2022-12-01T10:45:06.5722647Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5722970Z Ran 1 test in 7.111s
2022-12-01T10:45:06.5723137Z 
2022-12-01T10:45:06.5723237Z OK
2022-12-01T10:45:06.5723378Z 
2022-12-01T10:45:06.5723506Z Generating XML reports...
2022-12-01T10:45:06.5724113Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102340.xml
2022-12-01T10:45:06.5725037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5725503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5726105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5726575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5726813Z 
2022-12-01T10:45:06.5726926Z Running tests...
2022-12-01T10:45:06.5727341Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5728009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5728556Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5728849Z 
2022-12-01T10:45:06.5729123Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5729465Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5729635Z 
2022-12-01T10:45:06.5729731Z OK (skipped=1)
2022-12-01T10:45:06.5729892Z 
2022-12-01T10:45:06.5730132Z Generating XML reports...
2022-12-01T10:45:06.5730757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102350.xml
2022-12-01T10:45:06.5731479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5731922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5732508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5732989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5733223Z 
2022-12-01T10:45:06.5733318Z Running tests...
2022-12-01T10:45:06.5733739Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5734371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5734939Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T10:45:06.5735264Z 
2022-12-01T10:45:06.5735522Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5735858Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5736023Z 
2022-12-01T10:45:06.5736135Z OK (skipped=1)
2022-12-01T10:45:06.5736295Z 
2022-12-01T10:45:06.5736403Z Generating XML reports...
2022-12-01T10:45:06.5737017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102352.xml
2022-12-01T10:45:06.5737742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5738206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5738773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5739255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5739491Z 
2022-12-01T10:45:06.5739603Z Running tests...
2022-12-01T10:45:06.5740023Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5740544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5741096Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T10:45:06.5741424Z 
2022-12-01T10:45:06.5741694Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5742006Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5742173Z 
2022-12-01T10:45:06.5742285Z OK (skipped=1)
2022-12-01T10:45:06.5742510Z 
2022-12-01T10:45:06.5742639Z Generating XML reports...
2022-12-01T10:45:06.5743263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102354.xml
2022-12-01T10:45:06.5744049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5744509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5745279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5752469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5752749Z 
2022-12-01T10:45:06.5752862Z Running tests...
2022-12-01T10:45:06.5753282Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5753814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5754351Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T10:45:06.5754668Z 
2022-12-01T10:45:06.5754929Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5755245Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5755400Z 
2022-12-01T10:45:06.5755494Z OK (skipped=1)
2022-12-01T10:45:06.5755651Z 
2022-12-01T10:45:06.5755757Z Generating XML reports...
2022-12-01T10:45:06.5756348Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102356.xml
2022-12-01T10:45:06.5757052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5757486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5758046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5758620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5758860Z 
2022-12-01T10:45:06.5758960Z Running tests...
2022-12-01T10:45:06.5759352Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5759869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5760391Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T10:45:06.5760699Z 
2022-12-01T10:45:06.5760958Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5761261Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5761413Z 
2022-12-01T10:45:06.5761511Z OK (skipped=1)
2022-12-01T10:45:06.5761658Z 
2022-12-01T10:45:06.5761773Z Generating XML reports...
2022-12-01T10:45:06.5762360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102358.xml
2022-12-01T10:45:06.5763067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5784341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5784981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5785430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5785654Z 
2022-12-01T10:45:06.5785758Z Running tests...
2022-12-01T10:45:06.5786146Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5786453Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5786741Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-12-01T10:45:06.5786874Z 
2022-12-01T10:45:06.5787145Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5787255Z Ran 1 test in 0.003s
2022-12-01T10:45:06.5787275Z 
2022-12-01T10:45:06.5787377Z OK (skipped=1)
2022-12-01T10:45:06.5787396Z 
2022-12-01T10:45:06.5787510Z Generating XML reports...
2022-12-01T10:45:06.5787956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102400.xml
2022-12-01T10:45:06.5788322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5788491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5788861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5789045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5789065Z 
2022-12-01T10:45:06.5789167Z Running tests...
2022-12-01T10:45:06.5789741Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5790077Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5790339Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5790360Z 
2022-12-01T10:45:06.5790624Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5790721Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5790742Z 
2022-12-01T10:45:06.5790846Z OK (skipped=1)
2022-12-01T10:45:06.5790865Z 
2022-12-01T10:45:06.5790972Z Generating XML reports...
2022-12-01T10:45:06.5791417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102402.xml
2022-12-01T10:45:06.5791783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5791957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5792404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5792601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5792621Z 
2022-12-01T10:45:06.5792721Z Running tests...
2022-12-01T10:45:06.5792980Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5793282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5793530Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5793740Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8686
2022-12-01T10:45:06.5793950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8687
2022-12-01T10:45:06.5794311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5794490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5794862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5795044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5795400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5795568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5795932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5796113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5796349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5796666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5797061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5797452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5797678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5797902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5798238Z STAGE:2022-12-01 10:24:10 8687:8687 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5798551Z STAGE:2022-12-01 10:24:10 8686:8686 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5798882Z STAGE:2022-12-01 10:24:11 8686:8686 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5799209Z STAGE:2022-12-01 10:24:11 8687:8687 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5799527Z STAGE:2022-12-01 10:24:11 8687:8687 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5799859Z STAGE:2022-12-01 10:24:11 8687:8687 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5800166Z STAGE:2022-12-01 10:24:11 8686:8686 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5800499Z STAGE:2022-12-01 10:24:11 8686:8686 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5800604Z ok (7.741s)
2022-12-01T10:45:06.5800624Z 
2022-12-01T10:45:06.5800886Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5801000Z Ran 1 test in 7.741s
2022-12-01T10:45:06.5801019Z 
2022-12-01T10:45:06.5801114Z OK
2022-12-01T10:45:06.5801138Z 
2022-12-01T10:45:06.5801263Z Generating XML reports...
2022-12-01T10:45:06.5801769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102404.xml
2022-12-01T10:45:06.5802138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5802316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5802697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5802893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5802914Z 
2022-12-01T10:45:06.5803022Z Running tests...
2022-12-01T10:45:06.5803290Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5803603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5803877Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5804098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9154
2022-12-01T10:45:06.5804301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9155
2022-12-01T10:45:06.5804674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5804850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5805230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5805418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5805782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5805956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5806397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5806569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5806817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5807064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5807468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5807870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5808104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5808336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5808675Z STAGE:2022-12-01 10:24:19 9155:9155 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5809005Z STAGE:2022-12-01 10:24:20 9154:9154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5809320Z STAGE:2022-12-01 10:24:21 9155:9155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5809649Z STAGE:2022-12-01 10:24:21 9154:9154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5809971Z STAGE:2022-12-01 10:24:21 9155:9155 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5810302Z STAGE:2022-12-01 10:24:21 9155:9155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5810625Z STAGE:2022-12-01 10:24:21 9154:9154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5810958Z STAGE:2022-12-01 10:24:21 9154:9154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5811066Z ok (7.588s)
2022-12-01T10:45:06.5811133Z 
2022-12-01T10:45:06.5811410Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5811524Z Ran 1 test in 7.588s
2022-12-01T10:45:06.5811544Z 
2022-12-01T10:45:06.5811621Z OK
2022-12-01T10:45:06.5811639Z 
2022-12-01T10:45:06.5811762Z Generating XML reports...
2022-12-01T10:45:06.5812215Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102414.xml
2022-12-01T10:45:06.5812588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5812767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5813145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5813344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5813368Z 
2022-12-01T10:45:06.5813480Z Running tests...
2022-12-01T10:45:06.5813733Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5814052Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5814325Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5814344Z 
2022-12-01T10:45:06.5814606Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5814718Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5814738Z 
2022-12-01T10:45:06.5814845Z OK (skipped=1)
2022-12-01T10:45:06.5814864Z 
2022-12-01T10:45:06.5814986Z Generating XML reports...
2022-12-01T10:45:06.5815431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102424.xml
2022-12-01T10:45:06.5815803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5816022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5816405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5816598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5816619Z 
2022-12-01T10:45:06.5816728Z Running tests...
2022-12-01T10:45:06.5816989Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5817303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5817572Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5817592Z 
2022-12-01T10:45:06.5817858Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5817977Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5817996Z 
2022-12-01T10:45:06.5818089Z OK (skipped=1)
2022-12-01T10:45:06.5818109Z 
2022-12-01T10:45:06.5818233Z Generating XML reports...
2022-12-01T10:45:06.5818685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102426.xml
2022-12-01T10:45:06.5819062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5819239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5819620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5819812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5819833Z 
2022-12-01T10:45:06.5819941Z Running tests...
2022-12-01T10:45:06.5820188Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5820556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5820844Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5821063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9773
2022-12-01T10:45:06.5821280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9774
2022-12-01T10:45:06.5821654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5821832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5822211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5822403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5822748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5822930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5823308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5823497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5823745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5823992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5824395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5824798Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5825086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5825301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5825641Z STAGE:2022-12-01 10:24:33 9773:9773 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5825969Z STAGE:2022-12-01 10:24:33 9774:9774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5826302Z STAGE:2022-12-01 10:24:35 9773:9773 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5826625Z STAGE:2022-12-01 10:24:35 9774:9774 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5826731Z ok (7.797s)
2022-12-01T10:45:06.5826751Z 
2022-12-01T10:45:06.5827017Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5827135Z Ran 1 test in 7.797s
2022-12-01T10:45:06.5827154Z 
2022-12-01T10:45:06.5827230Z OK
2022-12-01T10:45:06.5827271Z 
2022-12-01T10:45:06.5827378Z Generating XML reports...
2022-12-01T10:45:06.5827835Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102428.xml
2022-12-01T10:45:06.5828208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5828385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5828764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5828957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5828977Z 
2022-12-01T10:45:06.5829086Z Running tests...
2022-12-01T10:45:06.5829539Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5829862Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5830223Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5830455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10231
2022-12-01T10:45:06.5830674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10232
2022-12-01T10:45:06.5831052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5831229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5831606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5831798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5832150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5832323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5832704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5832894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5833140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5833385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5833787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5834187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5834420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5834631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5835065Z STAGE:2022-12-01 10:24:44 10231:10231 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5835397Z STAGE:2022-12-01 10:24:44 10232:10232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5835735Z STAGE:2022-12-01 10:24:45 10231:10231 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5836063Z STAGE:2022-12-01 10:24:45 10232:10232 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5836167Z ok (7.802s)
2022-12-01T10:45:06.5836189Z 
2022-12-01T10:45:06.5836453Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5836567Z Ran 1 test in 7.802s
2022-12-01T10:45:06.5836586Z 
2022-12-01T10:45:06.5836681Z OK
2022-12-01T10:45:06.5836701Z 
2022-12-01T10:45:06.5836808Z Generating XML reports...
2022-12-01T10:45:06.5837263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102438.xml
2022-12-01T10:45:06.5837647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5837824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5838202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5838393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5838412Z 
2022-12-01T10:45:06.5838522Z Running tests...
2022-12-01T10:45:06.5838787Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5839087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5839348Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5840158Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.688s)
2022-12-01T10:45:06.5840180Z 
2022-12-01T10:45:06.5840453Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5840566Z Ran 1 test in 1.689s
2022-12-01T10:45:06.5840586Z 
2022-12-01T10:45:06.5840693Z OK (skipped=1)
2022-12-01T10:45:06.5840712Z 
2022-12-01T10:45:06.5840836Z Generating XML reports...
2022-12-01T10:45:06.5841282Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102448.xml
2022-12-01T10:45:06.5841656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5841833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5842205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5842399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5842419Z 
2022-12-01T10:45:06.5842529Z Running tests...
2022-12-01T10:45:06.5842793Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5843109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5843382Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5844130Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.770s)
2022-12-01T10:45:06.5844199Z 
2022-12-01T10:45:06.5844476Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5844590Z Ran 1 test in 1.771s
2022-12-01T10:45:06.5844609Z 
2022-12-01T10:45:06.5844716Z OK (skipped=1)
2022-12-01T10:45:06.5844736Z 
2022-12-01T10:45:06.5844843Z Generating XML reports...
2022-12-01T10:45:06.5845301Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102452.xml
2022-12-01T10:45:06.5845677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5845854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5846234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5846429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5846453Z 
2022-12-01T10:45:06.5846562Z Running tests...
2022-12-01T10:45:06.5846835Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5847138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5847418Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5847647Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11059
2022-12-01T10:45:06.5847865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11060
2022-12-01T10:45:06.5848238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5848416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5848796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5849039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5849421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5849578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5849956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5850145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5850391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5850635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5851033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5851443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5851677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5851891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5851993Z ok (6.799s)
2022-12-01T10:45:06.5852013Z 
2022-12-01T10:45:06.5852282Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5852398Z Ran 1 test in 6.799s
2022-12-01T10:45:06.5852418Z 
2022-12-01T10:45:06.5852512Z OK
2022-12-01T10:45:06.5852531Z 
2022-12-01T10:45:06.5852654Z Generating XML reports...
2022-12-01T10:45:06.5853112Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102456.xml
2022-12-01T10:45:06.5853486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5853723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5854095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5854290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5854310Z 
2022-12-01T10:45:06.5854418Z Running tests...
2022-12-01T10:45:06.5854682Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5854994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5855266Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5855485Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11436
2022-12-01T10:45:06.5855704Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11437
2022-12-01T10:45:06.5856071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5856250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5856632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5856824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5857193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5857368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5857741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5857931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5858179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5858471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5858889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5859285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5859518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5859748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5859987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.5860229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.5860636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.5861035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.5861260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.5861658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.5861894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.5862284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.5862524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-12-01T10:45:06.5862817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-12-01T10:45:06.5863213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T10:45:06.5863608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T10:45:06.5863712Z ok (8.544s)
2022-12-01T10:45:06.5863732Z 
2022-12-01T10:45:06.5863984Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5864101Z Ran 1 test in 8.545s
2022-12-01T10:45:06.5864120Z 
2022-12-01T10:45:06.5864213Z OK
2022-12-01T10:45:06.5864233Z 
2022-12-01T10:45:06.5864357Z Generating XML reports...
2022-12-01T10:45:06.5864813Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102505.xml
2022-12-01T10:45:06.5865187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5865372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5865757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5865950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5865970Z 
2022-12-01T10:45:06.5866061Z Running tests...
2022-12-01T10:45:06.5866328Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5866647Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5866906Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5867127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11941
2022-12-01T10:45:06.5867345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11942
2022-12-01T10:45:06.5867769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5867953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5868322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5868517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5868887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5869061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5869662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5869862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5870112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5870360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5870772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5871153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5871388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5871617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5871722Z ok (7.038s)
2022-12-01T10:45:06.5871742Z 
2022-12-01T10:45:06.5872008Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5872121Z Ran 1 test in 7.038s
2022-12-01T10:45:06.5872249Z 
2022-12-01T10:45:06.5872351Z OK
2022-12-01T10:45:06.5872371Z 
2022-12-01T10:45:06.5872501Z Generating XML reports...
2022-12-01T10:45:06.5872940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102516.xml
2022-12-01T10:45:06.5873317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5873495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5873877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5874070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5874090Z 
2022-12-01T10:45:06.5874198Z Running tests...
2022-12-01T10:45:06.5874460Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5874776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5875209Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5875230Z 
2022-12-01T10:45:06.5875477Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5875590Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5875609Z 
2022-12-01T10:45:06.5875714Z OK (skipped=1)
2022-12-01T10:45:06.5875734Z 
2022-12-01T10:45:06.5875857Z Generating XML reports...
2022-12-01T10:45:06.5876308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102525.xml
2022-12-01T10:45:06.5876681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5876860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5877241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5877498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5877519Z 
2022-12-01T10:45:06.5877618Z Running tests...
2022-12-01T10:45:06.5877887Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5878203Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5878624Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5878644Z 
2022-12-01T10:45:06.5878908Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5879069Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5879090Z 
2022-12-01T10:45:06.5879202Z OK (skipped=1)
2022-12-01T10:45:06.5879221Z 
2022-12-01T10:45:06.5879345Z Generating XML reports...
2022-12-01T10:45:06.5879796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102527.xml
2022-12-01T10:45:06.5880159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5880335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5880718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5880910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5880930Z 
2022-12-01T10:45:06.5881039Z Running tests...
2022-12-01T10:45:06.5881302Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5881619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5882047Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5882120Z 
2022-12-01T10:45:06.5882393Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5882490Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5882510Z 
2022-12-01T10:45:06.5882616Z OK (skipped=1)
2022-12-01T10:45:06.5882636Z 
2022-12-01T10:45:06.5882759Z Generating XML reports...
2022-12-01T10:45:06.5883208Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102530.xml
2022-12-01T10:45:06.5883582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5883758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5884139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5884331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5884356Z 
2022-12-01T10:45:06.5884446Z Running tests...
2022-12-01T10:45:06.5884713Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5885027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5885446Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5885466Z 
2022-12-01T10:45:06.5885726Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5885838Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5885858Z 
2022-12-01T10:45:06.5885964Z OK (skipped=1)
2022-12-01T10:45:06.5885984Z 
2022-12-01T10:45:06.5886107Z Generating XML reports...
2022-12-01T10:45:06.5886553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102532.xml
2022-12-01T10:45:06.5886912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5887198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5887597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5887789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5887808Z 
2022-12-01T10:45:06.5887918Z Running tests...
2022-12-01T10:45:06.5888184Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5888499Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5888911Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5888932Z 
2022-12-01T10:45:06.5889198Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5889298Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5889317Z 
2022-12-01T10:45:06.5889431Z OK (skipped=1)
2022-12-01T10:45:06.5889451Z 
2022-12-01T10:45:06.5889575Z Generating XML reports...
2022-12-01T10:45:06.5890025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102534.xml
2022-12-01T10:45:06.5890396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5890573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5890956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5891151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5891171Z 
2022-12-01T10:45:06.5891278Z Running tests...
2022-12-01T10:45:06.5891524Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5891905Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5892316Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5892336Z 
2022-12-01T10:45:06.5892601Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5892712Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5892731Z 
2022-12-01T10:45:06.5892840Z OK (skipped=1)
2022-12-01T10:45:06.5892859Z 
2022-12-01T10:45:06.5892981Z Generating XML reports...
2022-12-01T10:45:06.5893431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102536.xml
2022-12-01T10:45:06.5893802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5893963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5894358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5894552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5894572Z 
2022-12-01T10:45:06.5894681Z Running tests...
2022-12-01T10:45:06.5894944Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5895258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5895678Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5895699Z 
2022-12-01T10:45:06.5895961Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5896055Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5896092Z 
2022-12-01T10:45:06.5896182Z OK (skipped=1)
2022-12-01T10:45:06.5896201Z 
2022-12-01T10:45:06.5896329Z Generating XML reports...
2022-12-01T10:45:06.5896821Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102538.xml
2022-12-01T10:45:06.5897202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5897377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5897753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5897947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5897967Z 
2022-12-01T10:45:06.5898079Z Running tests...
2022-12-01T10:45:06.5898325Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5898639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5899049Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5899076Z 
2022-12-01T10:45:06.5899339Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5899450Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5899469Z 
2022-12-01T10:45:06.5899576Z OK (skipped=1)
2022-12-01T10:45:06.5899595Z 
2022-12-01T10:45:06.5899720Z Generating XML reports...
2022-12-01T10:45:06.5900169Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102540.xml
2022-12-01T10:45:06.5900541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5900700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5901083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5901342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5901361Z 
2022-12-01T10:45:06.5901473Z Running tests...
2022-12-01T10:45:06.5901738Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5902054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5902452Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5902473Z 
2022-12-01T10:45:06.5902733Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5902843Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5902863Z 
2022-12-01T10:45:06.5902952Z OK (skipped=1)
2022-12-01T10:45:06.5902971Z 
2022-12-01T10:45:06.5903094Z Generating XML reports...
2022-12-01T10:45:06.5903541Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102542.xml
2022-12-01T10:45:06.5903922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5904095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5904476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5904667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5904687Z 
2022-12-01T10:45:06.5904795Z Running tests...
2022-12-01T10:45:06.5905040Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5905355Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5905660Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5905680Z 
2022-12-01T10:45:06.5905940Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5906059Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5906124Z 
2022-12-01T10:45:06.5906238Z OK (skipped=1)
2022-12-01T10:45:06.5906257Z 
2022-12-01T10:45:06.5906383Z Generating XML reports...
2022-12-01T10:45:06.5906834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102544.xml
2022-12-01T10:45:06.5907206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5907365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5907748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5907939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5907959Z 
2022-12-01T10:45:06.5908066Z Running tests...
2022-12-01T10:45:06.5908337Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5908654Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5909053Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5909074Z 
2022-12-01T10:45:06.5909333Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5909684Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5909705Z 
2022-12-01T10:45:06.5909797Z OK (skipped=1)
2022-12-01T10:45:06.5909816Z 
2022-12-01T10:45:06.5909942Z Generating XML reports...
2022-12-01T10:45:06.5910407Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102546.xml
2022-12-01T10:45:06.5910777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5911041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5911428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5911619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5911639Z 
2022-12-01T10:45:06.5911747Z Running tests...
2022-12-01T10:45:06.5912008Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5912306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5912718Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5912739Z 
2022-12-01T10:45:06.5912999Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5913110Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5913130Z 
2022-12-01T10:45:06.5913237Z OK (skipped=1)
2022-12-01T10:45:06.5913260Z 
2022-12-01T10:45:06.5913385Z Generating XML reports...
2022-12-01T10:45:06.5913838Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102548.xml
2022-12-01T10:45:06.5914209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5914387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5914751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5914944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5914964Z 
2022-12-01T10:45:06.5915071Z Running tests...
2022-12-01T10:45:06.5915334Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5915650Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5916121Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.5916153Z 
2022-12-01T10:45:06.5916645Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5916857Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5916887Z 
2022-12-01T10:45:06.5916994Z OK (skipped=1)
2022-12-01T10:45:06.5917029Z 
2022-12-01T10:45:06.5917137Z Generating XML reports...
2022-12-01T10:45:06.5917596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102551.xml
2022-12-01T10:45:06.5917965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5918141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5918520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5918720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5918740Z 
2022-12-01T10:45:06.5918849Z Running tests...
2022-12-01T10:45:06.5919112Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5919408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5919698Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5919718Z 
2022-12-01T10:45:06.5919981Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5920092Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5920111Z 
2022-12-01T10:45:06.5920218Z OK (skipped=1)
2022-12-01T10:45:06.5920237Z 
2022-12-01T10:45:06.5920361Z Generating XML reports...
2022-12-01T10:45:06.5920805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102553.xml
2022-12-01T10:45:06.5921253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5921431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5921789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5921981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5922001Z 
2022-12-01T10:45:06.5922108Z Running tests...
2022-12-01T10:45:06.5922367Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5922678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5922952Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5922972Z 
2022-12-01T10:45:06.5923240Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5923355Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5923374Z 
2022-12-01T10:45:06.5923482Z OK (skipped=1)
2022-12-01T10:45:06.5923501Z 
2022-12-01T10:45:06.5923606Z Generating XML reports...
2022-12-01T10:45:06.5924059Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102555.xml
2022-12-01T10:45:06.5924431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5924606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5924987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5925176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5925196Z 
2022-12-01T10:45:06.5925303Z Running tests...
2022-12-01T10:45:06.5925572Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5925916Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5926197Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5926217Z 
2022-12-01T10:45:06.5926481Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5926593Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5926612Z 
2022-12-01T10:45:06.5926717Z OK (skipped=1)
2022-12-01T10:45:06.5926736Z 
2022-12-01T10:45:06.5926858Z Generating XML reports...
2022-12-01T10:45:06.5927309Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102557.xml
2022-12-01T10:45:06.5927681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5927861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5928230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5928422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5928442Z 
2022-12-01T10:45:06.5928550Z Running tests...
2022-12-01T10:45:06.5928811Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5929127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5929406Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5929426Z 
2022-12-01T10:45:06.5929688Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5929800Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5929819Z 
2022-12-01T10:45:06.5929923Z OK (skipped=1)
2022-12-01T10:45:06.5929990Z 
2022-12-01T10:45:06.5930103Z Generating XML reports...
2022-12-01T10:45:06.5930557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102559.xml
2022-12-01T10:45:06.5930925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5931099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5931477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5931668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5931688Z 
2022-12-01T10:45:06.5931796Z Running tests...
2022-12-01T10:45:06.5932064Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5932360Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5932643Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5932662Z 
2022-12-01T10:45:06.5932926Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5933038Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5933057Z 
2022-12-01T10:45:06.5933165Z OK (skipped=1)
2022-12-01T10:45:06.5933184Z 
2022-12-01T10:45:06.5933306Z Generating XML reports...
2022-12-01T10:45:06.5933754Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102601.xml
2022-12-01T10:45:06.5934122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5934298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5934664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5934904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5934924Z 
2022-12-01T10:45:06.5935038Z Running tests...
2022-12-01T10:45:06.5935301Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5935615Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5935883Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5935903Z 
2022-12-01T10:45:06.5936163Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5936274Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5936293Z 
2022-12-01T10:45:06.5936398Z OK (skipped=1)
2022-12-01T10:45:06.5936417Z 
2022-12-01T10:45:06.5936521Z Generating XML reports...
2022-12-01T10:45:06.5936966Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102603.xml
2022-12-01T10:45:06.5937348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5937526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5937906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5938095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5938115Z 
2022-12-01T10:45:06.5938223Z Running tests...
2022-12-01T10:45:06.5938483Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5938796Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5939049Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5939069Z 
2022-12-01T10:45:06.5939388Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5939506Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5939526Z 
2022-12-01T10:45:06.5939637Z OK (skipped=1)
2022-12-01T10:45:06.5939656Z 
2022-12-01T10:45:06.5939782Z Generating XML reports...
2022-12-01T10:45:06.5940224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102605.xml
2022-12-01T10:45:06.5940594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5940766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5941127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5941319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5941339Z 
2022-12-01T10:45:06.5941446Z Running tests...
2022-12-01T10:45:06.5941711Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5942027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5942302Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5942321Z 
2022-12-01T10:45:06.5942580Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5942693Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5942712Z 
2022-12-01T10:45:06.5942820Z OK (skipped=1)
2022-12-01T10:45:06.5942839Z 
2022-12-01T10:45:06.5942944Z Generating XML reports...
2022-12-01T10:45:06.5943393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102607.xml
2022-12-01T10:45:06.5943767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5943946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5944371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5944574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5944594Z 
2022-12-01T10:45:06.5944702Z Running tests...
2022-12-01T10:45:06.5944966Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5945279Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5945530Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5945550Z 
2022-12-01T10:45:06.5945815Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5945931Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5945950Z 
2022-12-01T10:45:06.5946059Z OK (skipped=1)
2022-12-01T10:45:06.5946082Z 
2022-12-01T10:45:06.5946207Z Generating XML reports...
2022-12-01T10:45:06.5946661Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102609.xml
2022-12-01T10:45:06.5947031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5947207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5947584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5947759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5947778Z 
2022-12-01T10:45:06.5947887Z Running tests...
2022-12-01T10:45:06.5948153Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5948467Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5948781Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5948801Z 
2022-12-01T10:45:06.5949063Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5949173Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5949193Z 
2022-12-01T10:45:06.5949299Z OK (skipped=1)
2022-12-01T10:45:06.5949319Z 
2022-12-01T10:45:06.5949665Z Generating XML reports...
2022-12-01T10:45:06.5950135Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102611.xml
2022-12-01T10:45:06.5950504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5950678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5951056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5951254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5951274Z 
2022-12-01T10:45:06.5951386Z Running tests...
2022-12-01T10:45:06.5951646Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5951956Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5952199Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5952219Z 
2022-12-01T10:45:06.5952479Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5952591Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5952610Z 
2022-12-01T10:45:06.5952717Z OK (skipped=1)
2022-12-01T10:45:06.5952736Z 
2022-12-01T10:45:06.5952859Z Generating XML reports...
2022-12-01T10:45:06.5953305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102613.xml
2022-12-01T10:45:06.5953756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5953946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5954332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5954506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5954525Z 
2022-12-01T10:45:06.5954632Z Running tests...
2022-12-01T10:45:06.5954895Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5955212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5955494Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-12-01T10:45:06.5955514Z 
2022-12-01T10:45:06.5955777Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5955894Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5955914Z 
2022-12-01T10:45:06.5956023Z OK (skipped=1)
2022-12-01T10:45:06.5956042Z 
2022-12-01T10:45:06.5956165Z Generating XML reports...
2022-12-01T10:45:06.5956599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102615.xml
2022-12-01T10:45:06.5956968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5957142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5957524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5957717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5957737Z 
2022-12-01T10:45:06.5957843Z Running tests...
2022-12-01T10:45:06.5958109Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5958501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5958775Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-12-01T10:45:06.5958814Z 
2022-12-01T10:45:06.5959055Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5959166Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5959185Z 
2022-12-01T10:45:06.5959291Z OK (skipped=1)
2022-12-01T10:45:06.5959310Z 
2022-12-01T10:45:06.5959433Z Generating XML reports...
2022-12-01T10:45:06.5959878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102617.xml
2022-12-01T10:45:06.5960250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5960426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5960809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5960982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5961020Z 
2022-12-01T10:45:06.5961110Z Running tests...
2022-12-01T10:45:06.5961373Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5961687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5961953Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5961974Z 
2022-12-01T10:45:06.5962235Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5962347Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5962366Z 
2022-12-01T10:45:06.5962472Z OK (skipped=1)
2022-12-01T10:45:06.5962491Z 
2022-12-01T10:45:06.5962616Z Generating XML reports...
2022-12-01T10:45:06.5963090Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102620.xml
2022-12-01T10:45:06.5963470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5963646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5964022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5964213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5964233Z 
2022-12-01T10:45:06.5964342Z Running tests...
2022-12-01T10:45:06.5964604Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5964919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5965165Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5965392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15088
2022-12-01T10:45:06.5965611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15089
2022-12-01T10:45:06.5965983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5966158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5966536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5966730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5967095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5967269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5967696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5967887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5968132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5968379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5968779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5969178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5969408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5969637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5969726Z ok (6.866s)
2022-12-01T10:45:06.5969763Z 
2022-12-01T10:45:06.5970014Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5970128Z Ran 1 test in 6.866s
2022-12-01T10:45:06.5970148Z 
2022-12-01T10:45:06.5970240Z OK
2022-12-01T10:45:06.5970259Z 
2022-12-01T10:45:06.5970382Z Generating XML reports...
2022-12-01T10:45:06.5970834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102622.xml
2022-12-01T10:45:06.5971206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5971385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5971770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5971944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5971989Z 
2022-12-01T10:45:06.5972079Z Running tests...
2022-12-01T10:45:06.5972387Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5972711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5972972Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5972992Z 
2022-12-01T10:45:06.5973252Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5973364Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5973383Z 
2022-12-01T10:45:06.5973490Z OK (skipped=1)
2022-12-01T10:45:06.5973509Z 
2022-12-01T10:45:06.5973633Z Generating XML reports...
2022-12-01T10:45:06.5974064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102631.xml
2022-12-01T10:45:06.5974434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5974616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5974993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5975182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5975201Z 
2022-12-01T10:45:06.5975308Z Running tests...
2022-12-01T10:45:06.5975571Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5975887Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5976136Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5976172Z 
2022-12-01T10:45:06.5976414Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5976579Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5976598Z 
2022-12-01T10:45:06.5976706Z OK (skipped=1)
2022-12-01T10:45:06.5976728Z 
2022-12-01T10:45:06.5976850Z Generating XML reports...
2022-12-01T10:45:06.5977296Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102633.xml
2022-12-01T10:45:06.5977664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5977840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5978214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5978384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5978421Z 
2022-12-01T10:45:06.5978510Z Running tests...
2022-12-01T10:45:06.5978771Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5979135Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5979413Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.5979432Z 
2022-12-01T10:45:06.5979694Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5979807Z Ran 1 test in 0.002s
2022-12-01T10:45:06.5979827Z 
2022-12-01T10:45:06.5979933Z OK (skipped=1)
2022-12-01T10:45:06.5979952Z 
2022-12-01T10:45:06.5980075Z Generating XML reports...
2022-12-01T10:45:06.5980507Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102635.xml
2022-12-01T10:45:06.5980877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5981052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5981430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5981672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5981694Z 
2022-12-01T10:45:06.5981807Z Running tests...
2022-12-01T10:45:06.5982072Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5982384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5982626Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5982845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15761
2022-12-01T10:45:06.5983058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15762
2022-12-01T10:45:06.5983424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5983603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5983984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5984174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5984538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5984712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5985069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5985257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5985505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5985747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5986208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5986607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5986839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5987071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5987414Z STAGE:2022-12-01 10:26:42 15762:15762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5987727Z STAGE:2022-12-01 10:26:42 15761:15761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5988066Z STAGE:2022-12-01 10:26:42 15762:15762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5988394Z STAGE:2022-12-01 10:26:42 15761:15761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5988732Z STAGE:2022-12-01 10:26:43 15761:15761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5989053Z STAGE:2022-12-01 10:26:43 15762:15762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5989623Z STAGE:2022-12-01 10:26:43 15761:15761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5989980Z STAGE:2022-12-01 10:26:43 15762:15762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5990308Z STAGE:2022-12-01 10:26:43 15761:15761 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5990629Z STAGE:2022-12-01 10:26:43 15762:15762 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5990945Z STAGE:2022-12-01 10:26:43 15761:15761 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5991269Z STAGE:2022-12-01 10:26:43 15762:15762 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.5991442Z ok (6.204s)
2022-12-01T10:45:06.5991464Z 
2022-12-01T10:45:06.5991739Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5991850Z Ran 1 test in 6.204s
2022-12-01T10:45:06.5991870Z 
2022-12-01T10:45:06.5991963Z OK
2022-12-01T10:45:06.5991984Z 
2022-12-01T10:45:06.5992107Z Generating XML reports...
2022-12-01T10:45:06.5992561Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102637.xml
2022-12-01T10:45:06.5992915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5993095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5993474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5993672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5993691Z 
2022-12-01T10:45:06.5993801Z Running tests...
2022-12-01T10:45:06.5994064Z ----------------------------------------------------------------------
2022-12-01T10:45:06.5994378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.5994641Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.5994858Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16138
2022-12-01T10:45:06.5995058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16139
2022-12-01T10:45:06.5995423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5995595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5995971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5996227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5996587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.5996756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.5997128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.5997301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.5997543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.5997779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.5998178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5998580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.5998805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.5999027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.5999358Z STAGE:2022-12-01 10:26:51 16139:16139 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.5999680Z STAGE:2022-12-01 10:26:51 16138:16138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6000213Z STAGE:2022-12-01 10:26:51 16139:16139 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:51 16138:16138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6000245Z 
2022-12-01T10:45:06.6000559Z STAGE:2022-12-01 10:26:51 16139:16139 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6000925Z STAGE:2022-12-01 10:26:51 16138:16138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6001270Z STAGE:2022-12-01 10:26:51 16139:16139 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6001596Z STAGE:2022-12-01 10:26:51 16138:16138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6001919Z STAGE:2022-12-01 10:26:51 16139:16139 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6002239Z STAGE:2022-12-01 10:26:51 16138:16138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6002569Z STAGE:2022-12-01 10:26:51 16139:16139 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6002892Z STAGE:2022-12-01 10:26:51 16138:16138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6002995Z ok (6.238s)
2022-12-01T10:45:06.6003015Z 
2022-12-01T10:45:06.6003266Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6003377Z Ran 1 test in 6.239s
2022-12-01T10:45:06.6003396Z 
2022-12-01T10:45:06.6003487Z OK
2022-12-01T10:45:06.6003506Z 
2022-12-01T10:45:06.6003625Z Generating XML reports...
2022-12-01T10:45:06.6004077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102645.xml
2022-12-01T10:45:06.6004450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6004626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6005007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6005180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6005214Z 
2022-12-01T10:45:06.6005356Z Running tests...
2022-12-01T10:45:06.6005627Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6005942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6006214Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6006430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16558
2022-12-01T10:45:06.6006636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16559
2022-12-01T10:45:06.6007003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6007175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6007538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6007734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6008101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6008277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6008655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6008844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6009090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6009331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6009713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6010113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6010392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6010629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6010970Z STAGE:2022-12-01 10:26:59 16558:16558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6011296Z STAGE:2022-12-01 10:26:59 16559:16559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6011633Z STAGE:2022-12-01 10:26:59 16558:16558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6011961Z STAGE:2022-12-01 10:26:59 16559:16559 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6012279Z STAGE:2022-12-01 10:26:59 16558:16558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6012594Z STAGE:2022-12-01 10:26:59 16559:16559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6012932Z STAGE:2022-12-01 10:26:59 16558:16558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6013255Z STAGE:2022-12-01 10:26:59 16559:16559 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6013578Z STAGE:2022-12-01 10:26:59 16558:16558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6013897Z STAGE:2022-12-01 10:26:59 16559:16559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6014230Z STAGE:2022-12-01 10:26:59 16558:16558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6014557Z STAGE:2022-12-01 10:26:59 16559:16559 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6014658Z ok (6.100s)
2022-12-01T10:45:06.6014678Z 
2022-12-01T10:45:06.6014941Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6015092Z Ran 1 test in 6.100s
2022-12-01T10:45:06.6015112Z 
2022-12-01T10:45:06.6015209Z OK
2022-12-01T10:45:06.6015228Z 
2022-12-01T10:45:06.6015353Z Generating XML reports...
2022-12-01T10:45:06.6015808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102654.xml
2022-12-01T10:45:06.6016179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6016357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6016738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6016927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6016947Z 
2022-12-01T10:45:06.6017051Z Running tests...
2022-12-01T10:45:06.6017301Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6017617Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6017860Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:45:06.6017880Z 
2022-12-01T10:45:06.6018141Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6018252Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6018271Z 
2022-12-01T10:45:06.6018375Z OK (skipped=1)
2022-12-01T10:45:06.6018394Z 
2022-12-01T10:45:06.6018514Z Generating XML reports...
2022-12-01T10:45:06.6018959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102702.xml
2022-12-01T10:45:06.6019314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6019488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6019943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6020142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6020161Z 
2022-12-01T10:45:06.6020262Z Running tests...
2022-12-01T10:45:06.6020520Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6020828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6021077Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:45:06.6021097Z 
2022-12-01T10:45:06.6021358Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6021452Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6021471Z 
2022-12-01T10:45:06.6021570Z OK (skipped=1)
2022-12-01T10:45:06.6021589Z 
2022-12-01T10:45:06.6021704Z Generating XML reports...
2022-12-01T10:45:06.6022161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102704.xml
2022-12-01T10:45:06.6022530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6022704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6023078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6023265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6023285Z 
2022-12-01T10:45:06.6023389Z Running tests...
2022-12-01T10:45:06.6023636Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6023948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6024194Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6024463Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17091
2022-12-01T10:45:06.6024675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17092
2022-12-01T10:45:06.6025046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6025215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6025591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6025764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6026131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6026304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6026676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6026864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6027109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6027351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6027744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6028139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6028353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6028579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6028681Z ok (6.693s)
2022-12-01T10:45:06.6028701Z 
2022-12-01T10:45:06.6029003Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6029123Z Ran 1 test in 6.693s
2022-12-01T10:45:06.6029142Z 
2022-12-01T10:45:06.6029233Z OK
2022-12-01T10:45:06.6029253Z 
2022-12-01T10:45:06.6029838Z Generating XML reports...
2022-12-01T10:45:06.6030328Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102706.xml
2022-12-01T10:45:06.6030684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6030862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6031244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6031433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6031453Z 
2022-12-01T10:45:06.6031563Z Running tests...
2022-12-01T10:45:06.6031825Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6032139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6032404Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6032623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17453
2022-12-01T10:45:06.6032827Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17454
2022-12-01T10:45:06.6033197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6033373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6033752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6034056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6034427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6034596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6034970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6035140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6035385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6035628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6036032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6036432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6036671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6036901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6037004Z ok (6.960s)
2022-12-01T10:45:06.6037024Z 
2022-12-01T10:45:06.6037291Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6037385Z Ran 1 test in 6.960s
2022-12-01T10:45:06.6037405Z 
2022-12-01T10:45:06.6037493Z OK
2022-12-01T10:45:06.6037513Z 
2022-12-01T10:45:06.6037631Z Generating XML reports...
2022-12-01T10:45:06.6038079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102715.xml
2022-12-01T10:45:06.6038448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6038628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6039075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6039280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6039300Z 
2022-12-01T10:45:06.6039410Z Running tests...
2022-12-01T10:45:06.6039661Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6039981Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6040242Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:45:06.6040261Z 
2022-12-01T10:45:06.6040518Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6040629Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6040649Z 
2022-12-01T10:45:06.6040756Z OK (skipped=1)
2022-12-01T10:45:06.6040780Z 
2022-12-01T10:45:06.6040904Z Generating XML reports...
2022-12-01T10:45:06.6041358Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102724.xml
2022-12-01T10:45:06.6041718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6041893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6042277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6042468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6042487Z 
2022-12-01T10:45:06.6042594Z Running tests...
2022-12-01T10:45:06.6042849Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6043164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6043491Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6043711Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18025
2022-12-01T10:45:06.6043913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18026
2022-12-01T10:45:06.6044287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6044462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6044842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6045032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6045390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6045567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6045945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6046118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6046362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6046605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6047005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6047403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6047634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6047879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6048146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6048390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6048772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6049168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6049267Z ok (6.876s)
2022-12-01T10:45:06.6049287Z 
2022-12-01T10:45:06.6049557Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6049669Z Ran 1 test in 6.876s
2022-12-01T10:45:06.6049688Z 
2022-12-01T10:45:06.6049780Z OK
2022-12-01T10:45:06.6049799Z 
2022-12-01T10:45:06.6049917Z Generating XML reports...
2022-12-01T10:45:06.6050379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102726.xml
2022-12-01T10:45:06.6050753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6050912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6051297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6051486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6051506Z 
2022-12-01T10:45:06.6051614Z Running tests...
2022-12-01T10:45:06.6051877Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6052185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6052438Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:45:06.6052507Z 
2022-12-01T10:45:06.6052780Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6052888Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6052908Z 
2022-12-01T10:45:06.6052998Z OK (skipped=1)
2022-12-01T10:45:06.6053017Z 
2022-12-01T10:45:06.6053141Z Generating XML reports...
2022-12-01T10:45:06.6053590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102735.xml
2022-12-01T10:45:06.6053960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6054134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6054514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6054703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6054726Z 
2022-12-01T10:45:06.6054835Z Running tests...
2022-12-01T10:45:06.6055084Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6055398Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6055660Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6055882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18524
2022-12-01T10:45:06.6056098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18525
2022-12-01T10:45:06.6056467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6056639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6057019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6057258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6057622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6057795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6058167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6058356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6058599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6058841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6059243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6059650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6059880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6060091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6060253Z skip: Skipped due to small world size. (3.912s)
2022-12-01T10:45:06.6060273Z 
2022-12-01T10:45:06.6060539Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6060654Z Ran 1 test in 3.913s
2022-12-01T10:45:06.6060673Z 
2022-12-01T10:45:06.6060777Z OK (skipped=1)
2022-12-01T10:45:06.6060796Z 
2022-12-01T10:45:06.6060920Z Generating XML reports...
2022-12-01T10:45:06.6061368Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102737.xml
2022-12-01T10:45:06.6061740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6061955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6062339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6062528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6062547Z 
2022-12-01T10:45:06.6062656Z Running tests...
2022-12-01T10:45:06.6062919Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6063230Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6063512Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6063532Z 
2022-12-01T10:45:06.6063789Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6063903Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6063923Z 
2022-12-01T10:45:06.6064016Z OK (skipped=1)
2022-12-01T10:45:06.6064034Z 
2022-12-01T10:45:06.6064155Z Generating XML reports...
2022-12-01T10:45:06.6064600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102744.xml
2022-12-01T10:45:06.6064968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6065144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6065521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6065712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6065732Z 
2022-12-01T10:45:06.6065839Z Running tests...
2022-12-01T10:45:06.6066101Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6066445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6066751Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6066771Z 
2022-12-01T10:45:06.6067035Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6067146Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6067165Z 
2022-12-01T10:45:06.6067274Z OK (skipped=1)
2022-12-01T10:45:06.6067293Z 
2022-12-01T10:45:06.6067415Z Generating XML reports...
2022-12-01T10:45:06.6067867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102746.xml
2022-12-01T10:45:06.6068236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6068394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6068781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6068974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6068993Z 
2022-12-01T10:45:06.6069099Z Running tests...
2022-12-01T10:45:06.6069578Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6069917Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6070199Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6070957Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.703s)
2022-12-01T10:45:06.6071067Z 
2022-12-01T10:45:06.6071341Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6071454Z Ran 1 test in 1.703s
2022-12-01T10:45:06.6071474Z 
2022-12-01T10:45:06.6071563Z OK (skipped=1)
2022-12-01T10:45:06.6071582Z 
2022-12-01T10:45:06.6071706Z Generating XML reports...
2022-12-01T10:45:06.6072157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102748.xml
2022-12-01T10:45:06.6072528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6072706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6073088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6073280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6073303Z 
2022-12-01T10:45:06.6073410Z Running tests...
2022-12-01T10:45:06.6073681Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6073977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6074267Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6074488Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19187
2022-12-01T10:45:06.6074706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19188
2022-12-01T10:45:06.6075079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6075250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6075631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6075824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6076227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6076411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6076788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6076979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6077223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6077468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6077867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6078270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6078501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6078713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6079052Z STAGE:2022-12-01 10:27:57 19188:19188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6079430Z STAGE:2022-12-01 10:27:57 19187:19187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6079765Z STAGE:2022-12-01 10:27:59 19187:19187 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6080100Z STAGE:2022-12-01 10:27:59 19188:19188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6080206Z ok (7.676s)
2022-12-01T10:45:06.6080226Z 
2022-12-01T10:45:06.6080490Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6080662Z Ran 1 test in 7.676s
2022-12-01T10:45:06.6080685Z 
2022-12-01T10:45:06.6080777Z OK
2022-12-01T10:45:06.6080796Z 
2022-12-01T10:45:06.6080901Z Generating XML reports...
2022-12-01T10:45:06.6081355Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102752.xml
2022-12-01T10:45:06.6081727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6081898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6082276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6082468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6082488Z 
2022-12-01T10:45:06.6082591Z Running tests...
2022-12-01T10:45:06.6082853Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6083161Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6083462Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6083483Z 
2022-12-01T10:45:06.6083741Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6083848Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6083867Z 
2022-12-01T10:45:06.6083972Z OK (skipped=1)
2022-12-01T10:45:06.6083990Z 
2022-12-01T10:45:06.6084161Z Generating XML reports...
2022-12-01T10:45:06.6084619Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102802.xml
2022-12-01T10:45:06.6084988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6085164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6085578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6085774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6085794Z 
2022-12-01T10:45:06.6085902Z Running tests...
2022-12-01T10:45:06.6086170Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6086484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6086778Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6087524Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-12-01T10:45:06.6087550Z 
2022-12-01T10:45:06.6087815Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6087927Z Ran 1 test in 1.640s
2022-12-01T10:45:06.6087946Z 
2022-12-01T10:45:06.6088053Z OK (skipped=1)
2022-12-01T10:45:06.6088072Z 
2022-12-01T10:45:06.6088178Z Generating XML reports...
2022-12-01T10:45:06.6088626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102804.xml
2022-12-01T10:45:06.6089000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6089175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6089556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6089748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6089816Z 
2022-12-01T10:45:06.6089927Z Running tests...
2022-12-01T10:45:06.6090194Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6090490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6090784Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6090804Z 
2022-12-01T10:45:06.6091064Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6091173Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6091192Z 
2022-12-01T10:45:06.6091296Z OK (skipped=1)
2022-12-01T10:45:06.6091314Z 
2022-12-01T10:45:06.6091435Z Generating XML reports...
2022-12-01T10:45:06.6091880Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102808.xml
2022-12-01T10:45:06.6092254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6092437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6092804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6092996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6093016Z 
2022-12-01T10:45:06.6093122Z Running tests...
2022-12-01T10:45:06.6093384Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6093697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6093982Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6094203Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19980
2022-12-01T10:45:06.6094424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19981
2022-12-01T10:45:06.6094839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6095003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6095386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6095576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6095939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6096112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6096481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6096670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6096924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6097154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6097554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6097954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6098183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6098409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6098568Z skip: Skipped due to small world size. (3.976s)
2022-12-01T10:45:06.6098588Z 
2022-12-01T10:45:06.6098852Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6099031Z Ran 1 test in 3.976s
2022-12-01T10:45:06.6099050Z 
2022-12-01T10:45:06.6099160Z OK (skipped=1)
2022-12-01T10:45:06.6099179Z 
2022-12-01T10:45:06.6099285Z Generating XML reports...
2022-12-01T10:45:06.6099743Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102810.xml
2022-12-01T10:45:06.6100112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6100289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6100666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6100856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6100876Z 
2022-12-01T10:45:06.6100982Z Running tests...
2022-12-01T10:45:06.6101248Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6101569Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6101840Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6101878Z 
2022-12-01T10:45:06.6102123Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6102235Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6102255Z 
2022-12-01T10:45:06.6102360Z OK (skipped=1)
2022-12-01T10:45:06.6102379Z 
2022-12-01T10:45:06.6102501Z Generating XML reports...
2022-12-01T10:45:06.6102948Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102816.xml
2022-12-01T10:45:06.6103320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6103493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6103922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6104103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6104123Z 
2022-12-01T10:45:06.6104231Z Running tests...
2022-12-01T10:45:06.6104493Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6104803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6105102Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6105122Z 
2022-12-01T10:45:06.6105382Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6105494Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6105513Z 
2022-12-01T10:45:06.6105620Z OK (skipped=1)
2022-12-01T10:45:06.6105643Z 
2022-12-01T10:45:06.6105764Z Generating XML reports...
2022-12-01T10:45:06.6106201Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102818.xml
2022-12-01T10:45:06.6106575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6106748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6107125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6107317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6107337Z 
2022-12-01T10:45:06.6107445Z Running tests...
2022-12-01T10:45:06.6107706Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6108022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6108344Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6108563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20463
2022-12-01T10:45:06.6108779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20464
2022-12-01T10:45:06.6109152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6109329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6109947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6110141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6110510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6110686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6111054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6111245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6111491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6111737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6112137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6112529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6112763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6112990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6113079Z ok (6.820s)
2022-12-01T10:45:06.6113189Z 
2022-12-01T10:45:06.6113447Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6113563Z Ran 1 test in 6.821s
2022-12-01T10:45:06.6113582Z 
2022-12-01T10:45:06.6113672Z OK
2022-12-01T10:45:06.6113692Z 
2022-12-01T10:45:06.6113817Z Generating XML reports...
2022-12-01T10:45:06.6114272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102820.xml
2022-12-01T10:45:06.6114646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6114824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6115204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6115378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6115417Z 
2022-12-01T10:45:06.6115511Z Running tests...
2022-12-01T10:45:06.6115775Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6116090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6116382Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6116601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20852
2022-12-01T10:45:06.6116821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20853
2022-12-01T10:45:06.6117192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6117368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6117734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6117996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6118363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6118538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6118912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6119099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6119344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6119585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6119969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6120376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6120610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6120839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6120941Z ok (6.912s)
2022-12-01T10:45:06.6120961Z 
2022-12-01T10:45:06.6121224Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6121337Z Ran 1 test in 6.912s
2022-12-01T10:45:06.6121357Z 
2022-12-01T10:45:06.6121448Z OK
2022-12-01T10:45:06.6121467Z 
2022-12-01T10:45:06.6121589Z Generating XML reports...
2022-12-01T10:45:06.6122023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102829.xml
2022-12-01T10:45:06.6122395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6122625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6123019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6123210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6123230Z 
2022-12-01T10:45:06.6123336Z Running tests...
2022-12-01T10:45:06.6123602Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6123919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6124204Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6124242Z 
2022-12-01T10:45:06.6124487Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6124607Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6124625Z 
2022-12-01T10:45:06.6124737Z OK (skipped=1)
2022-12-01T10:45:06.6124757Z 
2022-12-01T10:45:06.6124879Z Generating XML reports...
2022-12-01T10:45:06.6125335Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102838.xml
2022-12-01T10:45:06.6125706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6125879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6126261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6126435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6126474Z 
2022-12-01T10:45:06.6126565Z Running tests...
2022-12-01T10:45:06.6126828Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6127216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6127517Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6127738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21302
2022-12-01T10:45:06.6127956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21303
2022-12-01T10:45:06.6128333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6128509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6128872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6129064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6129438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6129615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6129989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6130184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6130432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6130679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6131064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6131464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6131746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6131997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6132224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6132468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6132866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6133262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6133363Z ok (7.005s)
2022-12-01T10:45:06.6133383Z 
2022-12-01T10:45:06.6133649Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6133749Z Ran 1 test in 7.006s
2022-12-01T10:45:06.6133769Z 
2022-12-01T10:45:06.6133860Z OK
2022-12-01T10:45:06.6133882Z 
2022-12-01T10:45:06.6134006Z Generating XML reports...
2022-12-01T10:45:06.6134461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102840.xml
2022-12-01T10:45:06.6134833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6135009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6135388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6135580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6135600Z 
2022-12-01T10:45:06.6135690Z Running tests...
2022-12-01T10:45:06.6135955Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6136330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6136629Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:45:06.6136649Z 
2022-12-01T10:45:06.6136911Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6137024Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6137043Z 
2022-12-01T10:45:06.6137152Z OK (skipped=1)
2022-12-01T10:45:06.6137171Z 
2022-12-01T10:45:06.6137293Z Generating XML reports...
2022-12-01T10:45:06.6137746Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102850.xml
2022-12-01T10:45:06.6138098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6138273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6138660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6138850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6138869Z 
2022-12-01T10:45:06.6138977Z Running tests...
2022-12-01T10:45:06.6139243Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6139557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6139846Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6140048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21757
2022-12-01T10:45:06.6140268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21758
2022-12-01T10:45:06.6140634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6140811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6141233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6141433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6141798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6141974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6142344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6142517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6142766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6143010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6143417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6143814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6144046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6144273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6144434Z skip: Skipped due to small world size. (3.950s)
2022-12-01T10:45:06.6144454Z 
2022-12-01T10:45:06.6144718Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6144813Z Ran 1 test in 3.950s
2022-12-01T10:45:06.6144833Z 
2022-12-01T10:45:06.6144938Z OK (skipped=1)
2022-12-01T10:45:06.6144957Z 
2022-12-01T10:45:06.6145079Z Generating XML reports...
2022-12-01T10:45:06.6145593Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102852.xml
2022-12-01T10:45:06.6145963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6146142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6146522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6146714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6146733Z 
2022-12-01T10:45:06.6146840Z Running tests...
2022-12-01T10:45:06.6147087Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6147399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6147662Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6147889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22059
2022-12-01T10:45:06.6148107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22060
2022-12-01T10:45:06.6148480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6148791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6149612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6149800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6150178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6150351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6150817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6151018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6151267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6151513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6151915Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6152314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6152527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6152753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6153001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6153240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6153636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6154027Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6154135Z ok (6.724s)
2022-12-01T10:45:06.6154156Z 
2022-12-01T10:45:06.6154421Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6154516Z Ran 1 test in 6.724s
2022-12-01T10:45:06.6154554Z 
2022-12-01T10:45:06.6154629Z OK
2022-12-01T10:45:06.6154648Z 
2022-12-01T10:45:06.6154770Z Generating XML reports...
2022-12-01T10:45:06.6155224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102858.xml
2022-12-01T10:45:06.6155676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6155853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6156235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6156431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6156451Z 
2022-12-01T10:45:06.6156560Z Running tests...
2022-12-01T10:45:06.6156810Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6157126Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6157386Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6157609Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22480
2022-12-01T10:45:06.6157829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22481
2022-12-01T10:45:06.6158204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6158380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6158760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6158934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6159296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6159469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6159842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6160079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6160334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6160580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6160981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6161375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6161586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6161813Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6161963Z skip: Need at least 3 CUDA devices (3.941s)
2022-12-01T10:45:06.6161987Z 
2022-12-01T10:45:06.6162257Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6162369Z Ran 1 test in 3.942s
2022-12-01T10:45:06.6162388Z 
2022-12-01T10:45:06.6162495Z OK (skipped=1)
2022-12-01T10:45:06.6162514Z 
2022-12-01T10:45:06.6162637Z Generating XML reports...
2022-12-01T10:45:06.6163089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102907.xml
2022-12-01T10:45:06.6163463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6163622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6164001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6164192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6164212Z 
2022-12-01T10:45:06.6164431Z Running tests...
2022-12-01T10:45:06.6164699Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6165014Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6165268Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-12-01T10:45:06.6165289Z 
2022-12-01T10:45:06.6165551Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6165648Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6165684Z 
2022-12-01T10:45:06.6165772Z OK (skipped=1)
2022-12-01T10:45:06.6165791Z 
2022-12-01T10:45:06.6165913Z Generating XML reports...
2022-12-01T10:45:06.6166365Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102913.xml
2022-12-01T10:45:06.6166740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6166918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6167298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6167488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6167508Z 
2022-12-01T10:45:06.6167616Z Running tests...
2022-12-01T10:45:06.6167866Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6168178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6168430Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-12-01T10:45:06.6168449Z 
2022-12-01T10:45:06.6168711Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6168821Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6168841Z 
2022-12-01T10:45:06.6168949Z OK (skipped=1)
2022-12-01T10:45:06.6168972Z 
2022-12-01T10:45:06.6169094Z Generating XML reports...
2022-12-01T10:45:06.6169590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102915.xml
2022-12-01T10:45:06.6169975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6170133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6170510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6170707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6170726Z 
2022-12-01T10:45:06.6170834Z Running tests...
2022-12-01T10:45:06.6171095Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6171407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6171668Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6171886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22933
2022-12-01T10:45:06.6172086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22934
2022-12-01T10:45:06.6172460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6172635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6173014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6173206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6173567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6173797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6174173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6174364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6174591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6174836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6175243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6175639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6175869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6176099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6176204Z ok (7.650s)
2022-12-01T10:45:06.6176224Z 
2022-12-01T10:45:06.6176491Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6176587Z Ran 1 test in 7.650s
2022-12-01T10:45:06.6176625Z 
2022-12-01T10:45:06.6176699Z OK
2022-12-01T10:45:06.6176719Z 
2022-12-01T10:45:06.6176844Z Generating XML reports...
2022-12-01T10:45:06.6177295Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102917.xml
2022-12-01T10:45:06.6177668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6177846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6178225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6178420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6178439Z 
2022-12-01T10:45:06.6178590Z Running tests...
2022-12-01T10:45:06.6178845Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6179208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6179478Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-12-01T10:45:06.6179498Z 
2022-12-01T10:45:06.6179761Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6179874Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6179893Z 
2022-12-01T10:45:06.6180001Z OK (skipped=1)
2022-12-01T10:45:06.6180021Z 
2022-12-01T10:45:06.6180143Z Generating XML reports...
2022-12-01T10:45:06.6180585Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102927.xml
2022-12-01T10:45:06.6180961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6181119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6181498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6181688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6181708Z 
2022-12-01T10:45:06.6181818Z Running tests...
2022-12-01T10:45:06.6182076Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6182388Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6182655Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6182876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23429
2022-12-01T10:45:06.6183130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23430
2022-12-01T10:45:06.6183508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6183684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6184062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6184253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6184614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6184788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6185162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6185350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6185583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6185827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6186227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6186625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6186858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6187088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6187252Z skip: Skipped due to small world size. (3.988s)
2022-12-01T10:45:06.6187271Z 
2022-12-01T10:45:06.6187535Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6187650Z Ran 1 test in 3.988s
2022-12-01T10:45:06.6187721Z 
2022-12-01T10:45:06.6187818Z OK (skipped=1)
2022-12-01T10:45:06.6187837Z 
2022-12-01T10:45:06.6187961Z Generating XML reports...
2022-12-01T10:45:06.6188417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102929.xml
2022-12-01T10:45:06.6188792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6188970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6189557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6189766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6189787Z 
2022-12-01T10:45:06.6189897Z Running tests...
2022-12-01T10:45:06.6190151Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6190478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6190739Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-12-01T10:45:06.6190760Z 
2022-12-01T10:45:06.6191022Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6191133Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6191153Z 
2022-12-01T10:45:06.6191262Z OK (skipped=1)
2022-12-01T10:45:06.6191282Z 
2022-12-01T10:45:06.6191405Z Generating XML reports...
2022-12-01T10:45:06.6191851Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102935.xml
2022-12-01T10:45:06.6192221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6192379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6192852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6193043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6193063Z 
2022-12-01T10:45:06.6193172Z Running tests...
2022-12-01T10:45:06.6193435Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6193751Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6194010Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6194231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23835
2022-12-01T10:45:06.6194448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23836
2022-12-01T10:45:06.6194800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6194985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6195365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6195556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6195920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6196098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6196472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6196660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6196887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6197132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6197591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6198006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6198237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6198463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6198624Z skip: Skipped due to small world size. (4.038s)
2022-12-01T10:45:06.6198644Z 
2022-12-01T10:45:06.6198912Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6199025Z Ran 1 test in 4.038s
2022-12-01T10:45:06.6199044Z 
2022-12-01T10:45:06.6199133Z OK (skipped=1)
2022-12-01T10:45:06.6199152Z 
2022-12-01T10:45:06.6199280Z Generating XML reports...
2022-12-01T10:45:06.6199736Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102937.xml
2022-12-01T10:45:06.6200110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6200289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6200671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6200864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6200883Z 
2022-12-01T10:45:06.6200991Z Running tests...
2022-12-01T10:45:06.6201254Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6201550Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6201831Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T10:45:06.6201901Z 
2022-12-01T10:45:06.6202173Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6202288Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6202307Z 
2022-12-01T10:45:06.6202416Z OK (skipped=1)
2022-12-01T10:45:06.6202435Z 
2022-12-01T10:45:06.6202558Z Generating XML reports...
2022-12-01T10:45:06.6203010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102944.xml
2022-12-01T10:45:06.6203384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6203541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6203919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6204113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6204133Z 
2022-12-01T10:45:06.6204242Z Running tests...
2022-12-01T10:45:06.6204504Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6204816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6205092Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T10:45:06.6205111Z 
2022-12-01T10:45:06.6205375Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6205487Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6205507Z 
2022-12-01T10:45:06.6205596Z OK (skipped=1)
2022-12-01T10:45:06.6205632Z 
2022-12-01T10:45:06.6205737Z Generating XML reports...
2022-12-01T10:45:06.6206187Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102946.xml
2022-12-01T10:45:06.6206605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6206786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6207170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6207364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6207384Z 
2022-12-01T10:45:06.6207493Z Running tests...
2022-12-01T10:45:06.6207755Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6208054Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6208329Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T10:45:06.6208349Z 
2022-12-01T10:45:06.6208608Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6208723Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6208743Z 
2022-12-01T10:45:06.6208853Z OK (skipped=1)
2022-12-01T10:45:06.6208872Z 
2022-12-01T10:45:06.6208996Z Generating XML reports...
2022-12-01T10:45:06.6209444Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102948.xml
2022-12-01T10:45:06.6209816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6209991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6210353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6210543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6210563Z 
2022-12-01T10:45:06.6210669Z Running tests...
2022-12-01T10:45:06.6210963Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6211655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6211939Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T10:45:06.6211959Z 
2022-12-01T10:45:06.6212226Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6212340Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6212359Z 
2022-12-01T10:45:06.6212449Z OK (skipped=1)
2022-12-01T10:45:06.6212487Z 
2022-12-01T10:45:06.6212591Z Generating XML reports...
2022-12-01T10:45:06.6213039Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102950.xml
2022-12-01T10:45:06.6213411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6213585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6213975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6214167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6214187Z 
2022-12-01T10:45:06.6214295Z Running tests...
2022-12-01T10:45:06.6214563Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6214860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6215122Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T10:45:06.6215142Z 
2022-12-01T10:45:06.6215402Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6215512Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6215531Z 
2022-12-01T10:45:06.6215637Z OK (skipped=1)
2022-12-01T10:45:06.6215656Z 
2022-12-01T10:45:06.6215778Z Generating XML reports...
2022-12-01T10:45:06.6216284Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102952.xml
2022-12-01T10:45:06.6216668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6216844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6217204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6217398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6217418Z 
2022-12-01T10:45:06.6217524Z Running tests...
2022-12-01T10:45:06.6217786Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6218096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6218375Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6218601Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24591
2022-12-01T10:45:06.6218818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24592
2022-12-01T10:45:06.6219173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6219349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6219729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6219918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6220281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6220457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6220899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6221089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6221337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6221567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6221975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6222378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6222610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6222841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6223089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6223336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6223734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6224131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6224362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.6224602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.6224997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6225442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6225552Z ok (4.248s)
2022-12-01T10:45:06.6225571Z 
2022-12-01T10:45:06.6225839Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6225954Z Ran 1 test in 4.248s
2022-12-01T10:45:06.6225973Z 
2022-12-01T10:45:06.6226069Z OK
2022-12-01T10:45:06.6226088Z 
2022-12-01T10:45:06.6226211Z Generating XML reports...
2022-12-01T10:45:06.6226645Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102954.xml
2022-12-01T10:45:06.6227016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6227195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6227578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6227776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6227795Z 
2022-12-01T10:45:06.6227903Z Running tests...
2022-12-01T10:45:06.6228166Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6228482Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6228733Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6228957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24921
2022-12-01T10:45:06.6229172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24922
2022-12-01T10:45:06.6229781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6229965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6230463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6230659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6231024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6231201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6231554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6231745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6231991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6232237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6232639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6233049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6233282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6233514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6233599Z ok (5.898s)
2022-12-01T10:45:06.6233635Z 
2022-12-01T10:45:06.6233883Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6233994Z Ran 1 test in 5.898s
2022-12-01T10:45:06.6234014Z 
2022-12-01T10:45:06.6234108Z OK
2022-12-01T10:45:06.6234127Z 
2022-12-01T10:45:06.6234252Z Generating XML reports...
2022-12-01T10:45:06.6234701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103001.xml
2022-12-01T10:45:06.6235138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6235323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6235705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6235879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6235918Z 
2022-12-01T10:45:06.6236009Z Running tests...
2022-12-01T10:45:06.6236270Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6236587Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6236868Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6237087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25201
2022-12-01T10:45:06.6237314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25202
2022-12-01T10:45:06.6237688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6237864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6238226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6238418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6238780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6238956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6239328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6239568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6239816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6240060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6240444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6240841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6241073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6241301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6241463Z skip: Skipped due to small world size. (3.850s)
2022-12-01T10:45:06.6241483Z 
2022-12-01T10:45:06.6241755Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6241871Z Ran 1 test in 3.850s
2022-12-01T10:45:06.6241891Z 
2022-12-01T10:45:06.6242003Z OK (skipped=1)
2022-12-01T10:45:06.6242021Z 
2022-12-01T10:45:06.6242145Z Generating XML reports...
2022-12-01T10:45:06.6242579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103009.xml
2022-12-01T10:45:06.6242949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6243126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6243505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6243696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6243716Z 
2022-12-01T10:45:06.6243823Z Running tests...
2022-12-01T10:45:06.6244093Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6244453Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6244712Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6244934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25487
2022-12-01T10:45:06.6245151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25488
2022-12-01T10:45:06.6245523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6245698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6246074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6246264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6246634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6246806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6247162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6247349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6247596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6247842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6248243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6248643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6248928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6249158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6249259Z ok (5.557s)
2022-12-01T10:45:06.6249279Z 
2022-12-01T10:45:06.6249529Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6249642Z Ran 1 test in 5.558s
2022-12-01T10:45:06.6249661Z 
2022-12-01T10:45:06.6249755Z OK
2022-12-01T10:45:06.6249774Z 
2022-12-01T10:45:06.6249899Z Generating XML reports...
2022-12-01T10:45:06.6250351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103015.xml
2022-12-01T10:45:06.6250728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6250903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6251299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6251474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6251511Z 
2022-12-01T10:45:06.6251602Z Running tests...
2022-12-01T10:45:06.6251864Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6252177Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6252450Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6252669Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25817
2022-12-01T10:45:06.6252891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25818
2022-12-01T10:45:06.6253259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6253502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6253881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6254073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6254440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6254612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6254986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6255177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6255425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6255674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6256061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6256462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6256693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6256923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6257026Z ok (4.027s)
2022-12-01T10:45:06.6257047Z 
2022-12-01T10:45:06.6257312Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6257423Z Ran 1 test in 4.027s
2022-12-01T10:45:06.6257443Z 
2022-12-01T10:45:06.6257536Z OK
2022-12-01T10:45:06.6257555Z 
2022-12-01T10:45:06.6257677Z Generating XML reports...
2022-12-01T10:45:06.6258176Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103023.xml
2022-12-01T10:45:06.6258549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6258723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6259098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6259287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6259306Z 
2022-12-01T10:45:06.6259415Z Running tests...
2022-12-01T10:45:06.6259675Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6259988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6260258Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6260486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26136
2022-12-01T10:45:06.6260703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26137
2022-12-01T10:45:06.6261074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6261248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6261625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6261817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6262178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6262354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6262758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6262959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6263206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6263447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6263847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6264239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6264471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6264702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6264808Z ok (5.784s)
2022-12-01T10:45:06.6264828Z 
2022-12-01T10:45:06.6265080Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6265196Z Ran 1 test in 5.784s
2022-12-01T10:45:06.6265215Z 
2022-12-01T10:45:06.6265311Z OK
2022-12-01T10:45:06.6265330Z 
2022-12-01T10:45:06.6265453Z Generating XML reports...
2022-12-01T10:45:06.6265907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103029.xml
2022-12-01T10:45:06.6266280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6266460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6266841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6267015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6267100Z 
2022-12-01T10:45:06.6267196Z Running tests...
2022-12-01T10:45:06.6267467Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6267783Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6268054Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6268271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26537
2022-12-01T10:45:06.6268489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26538
2022-12-01T10:45:06.6268864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6269038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6269620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6269831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6270206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6270383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6270758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6270947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6271195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6271442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6271823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6272301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6272543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6272770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6272873Z ok (5.796s)
2022-12-01T10:45:06.6272894Z 
2022-12-01T10:45:06.6273159Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6273273Z Ran 1 test in 5.796s
2022-12-01T10:45:06.6273293Z 
2022-12-01T10:45:06.6273387Z OK
2022-12-01T10:45:06.6273406Z 
2022-12-01T10:45:06.6273532Z Generating XML reports...
2022-12-01T10:45:06.6273964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103037.xml
2022-12-01T10:45:06.6274336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6274520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6274904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6275094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6275114Z 
2022-12-01T10:45:06.6275224Z Running tests...
2022-12-01T10:45:06.6275486Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6275800Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6276054Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6276275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26848
2022-12-01T10:45:06.6276494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26849
2022-12-01T10:45:06.6276943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6277121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6277501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6277691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6278057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6278233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6278590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6278780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6279032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6279328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6279733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6280132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6280361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6280588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6280689Z ok (4.081s)
2022-12-01T10:45:06.6280709Z 
2022-12-01T10:45:06.6280958Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6281072Z Ran 1 test in 4.081s
2022-12-01T10:45:06.6281095Z 
2022-12-01T10:45:06.6281188Z OK
2022-12-01T10:45:06.6281207Z 
2022-12-01T10:45:06.6281383Z Generating XML reports...
2022-12-01T10:45:06.6281846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103045.xml
2022-12-01T10:45:06.6282216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6282392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6282774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6282947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6282984Z 
2022-12-01T10:45:06.6283074Z Running tests...
2022-12-01T10:45:06.6283335Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6283648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6283911Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.6283930Z 
2022-12-01T10:45:06.6284197Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6284311Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6284330Z 
2022-12-01T10:45:06.6284437Z OK (skipped=1)
2022-12-01T10:45:06.6284456Z 
2022-12-01T10:45:06.6284578Z Generating XML reports...
2022-12-01T10:45:06.6285010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103052.xml
2022-12-01T10:45:06.6285380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6285554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6285935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6286184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6286204Z 
2022-12-01T10:45:06.6286315Z Running tests...
2022-12-01T10:45:06.6286580Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6286891Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6287144Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6287877Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.707s)
2022-12-01T10:45:06.6287916Z 
2022-12-01T10:45:06.6288162Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6288278Z Ran 1 test in 1.707s
2022-12-01T10:45:06.6288298Z 
2022-12-01T10:45:06.6288407Z OK (skipped=1)
2022-12-01T10:45:06.6288426Z 
2022-12-01T10:45:06.6288551Z Generating XML reports...
2022-12-01T10:45:06.6289001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103053.xml
2022-12-01T10:45:06.6289369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6289543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6289926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6290099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6290135Z 
2022-12-01T10:45:06.6290225Z Running tests...
2022-12-01T10:45:06.6290487Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6290849Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6291126Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.6291145Z 
2022-12-01T10:45:06.6291405Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6291518Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6291537Z 
2022-12-01T10:45:06.6291646Z OK (skipped=1)
2022-12-01T10:45:06.6291665Z 
2022-12-01T10:45:06.6291788Z Generating XML reports...
2022-12-01T10:45:06.6292217Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103057.xml
2022-12-01T10:45:06.6292584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6292759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6293145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6293337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6293356Z 
2022-12-01T10:45:06.6293465Z Running tests...
2022-12-01T10:45:06.6293734Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6294045Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6294308Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.6294329Z 
2022-12-01T10:45:06.6294570Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6294681Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6294700Z 
2022-12-01T10:45:06.6294806Z OK (skipped=1)
2022-12-01T10:45:06.6294825Z 
2022-12-01T10:45:06.6294948Z Generating XML reports...
2022-12-01T10:45:06.6295469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103059.xml
2022-12-01T10:45:06.6295843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6296020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6296398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6296573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6296610Z 
2022-12-01T10:45:06.6296701Z Running tests...
2022-12-01T10:45:06.6296959Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6297277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6297542Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-12-01T10:45:06.6297565Z 
2022-12-01T10:45:06.6297830Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6297942Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6297961Z 
2022-12-01T10:45:06.6298068Z OK (skipped=1)
2022-12-01T10:45:06.6298087Z 
2022-12-01T10:45:06.6298209Z Generating XML reports...
2022-12-01T10:45:06.6298639Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103102.xml
2022-12-01T10:45:06.6299008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6299181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6299562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6299753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6299777Z 
2022-12-01T10:45:06.6299886Z Running tests...
2022-12-01T10:45:06.6300196Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6300521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6300782Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6301509Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2022-12-01T10:45:06.6301548Z 
2022-12-01T10:45:06.6301792Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6301904Z Ran 1 test in 1.670s
2022-12-01T10:45:06.6301923Z 
2022-12-01T10:45:06.6302029Z OK (skipped=1)
2022-12-01T10:45:06.6302052Z 
2022-12-01T10:45:06.6302177Z Generating XML reports...
2022-12-01T10:45:06.6302626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103104.xml
2022-12-01T10:45:06.6302998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6303175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6303558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6303732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6303767Z 
2022-12-01T10:45:06.6303857Z Running tests...
2022-12-01T10:45:06.6304119Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6304430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6304809Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6305558Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.632s)
2022-12-01T10:45:06.6305579Z 
2022-12-01T10:45:06.6305842Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6305956Z Ran 1 test in 1.632s
2022-12-01T10:45:06.6305975Z 
2022-12-01T10:45:06.6306081Z OK (skipped=1)
2022-12-01T10:45:06.6306100Z 
2022-12-01T10:45:06.6306223Z Generating XML reports...
2022-12-01T10:45:06.6306653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103108.xml
2022-12-01T10:45:06.6307032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6307208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6307592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6307784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6307804Z 
2022-12-01T10:45:06.6307912Z Running tests...
2022-12-01T10:45:06.6308176Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6308489Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6308807Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6309856Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2022-12-01T10:45:06.6309903Z 
2022-12-01T10:45:06.6310171Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6310286Z Ran 1 test in 1.627s
2022-12-01T10:45:06.6310306Z 
2022-12-01T10:45:06.6310417Z OK (skipped=1)
2022-12-01T10:45:06.6310436Z 
2022-12-01T10:45:06.6310566Z Generating XML reports...
2022-12-01T10:45:06.6311011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103112.xml
2022-12-01T10:45:06.6311384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6311563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6311948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6312145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6312166Z 
2022-12-01T10:45:06.6312258Z Running tests...
2022-12-01T10:45:06.6312523Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6312840Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6313109Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6313332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28116
2022-12-01T10:45:06.6313551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28117
2022-12-01T10:45:06.6313925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6314169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6314538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6314728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6315094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6315268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6315644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6315831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6316079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6316323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6316731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6317114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6317347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6317575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6317834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbtcub4e8
2022-12-01T10:45:06.6318108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbtcub4e8/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6318365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpioxowdwh
2022-12-01T10:45:06.6318636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpioxowdwh/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6318921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6319141Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6319244Z ok (6.217s)
2022-12-01T10:45:06.6319264Z 
2022-12-01T10:45:06.6319537Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6319650Z Ran 1 test in 6.218s
2022-12-01T10:45:06.6319670Z 
2022-12-01T10:45:06.6319762Z OK
2022-12-01T10:45:06.6319781Z 
2022-12-01T10:45:06.6319906Z Generating XML reports...
2022-12-01T10:45:06.6320356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103116.xml
2022-12-01T10:45:06.6320731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6320907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6321279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6321472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6321491Z 
2022-12-01T10:45:06.6321599Z Running tests...
2022-12-01T10:45:06.6321864Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6322178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6322455Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6322677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28481
2022-12-01T10:45:06.6322893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28482
2022-12-01T10:45:06.6323247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6323482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6323862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6324055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6324421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6324595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6324976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6325167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6325412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6327731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6328142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6328541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6328772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6329001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6329258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj9ao7a13
2022-12-01T10:45:06.6329527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj9ao7a13/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6329781Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1pvqnq9
2022-12-01T10:45:06.6330104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1pvqnq9/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6330332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6330561Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6330794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6331026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6331128Z ok (6.102s)
2022-12-01T10:45:06.6331148Z 
2022-12-01T10:45:06.6331419Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6331535Z Ran 1 test in 6.102s
2022-12-01T10:45:06.6331555Z 
2022-12-01T10:45:06.6331650Z OK
2022-12-01T10:45:06.6331669Z 
2022-12-01T10:45:06.6331775Z Generating XML reports...
2022-12-01T10:45:06.6332239Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103124.xml
2022-12-01T10:45:06.6332608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6332786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6333173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6333364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6333384Z 
2022-12-01T10:45:06.6333492Z Running tests...
2022-12-01T10:45:06.6333755Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6334073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6334327Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6335137Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.594s)
2022-12-01T10:45:06.6335158Z 
2022-12-01T10:45:06.6335421Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6335534Z Ran 1 test in 1.594s
2022-12-01T10:45:06.6335554Z 
2022-12-01T10:45:06.6335662Z OK (skipped=1)
2022-12-01T10:45:06.6335680Z 
2022-12-01T10:45:06.6335805Z Generating XML reports...
2022-12-01T10:45:06.6336256Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103132.xml
2022-12-01T10:45:06.6336634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6336815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6337203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6337376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6337396Z 
2022-12-01T10:45:06.6337504Z Running tests...
2022-12-01T10:45:06.6337770Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6338085Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6338374Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6339123Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.729s)
2022-12-01T10:45:06.6339192Z 
2022-12-01T10:45:06.6339465Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6339579Z Ran 1 test in 1.730s
2022-12-01T10:45:06.6339598Z 
2022-12-01T10:45:06.6339708Z OK (skipped=1)
2022-12-01T10:45:06.6339726Z 
2022-12-01T10:45:06.6339832Z Generating XML reports...
2022-12-01T10:45:06.6340281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103136.xml
2022-12-01T10:45:06.6340647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6340826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6341211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6341405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6341428Z 
2022-12-01T10:45:06.6341543Z Running tests...
2022-12-01T10:45:06.6341810Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6342127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6342395Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6342617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29192
2022-12-01T10:45:06.6342837Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29193
2022-12-01T10:45:06.6343209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6343387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6343771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6344019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6344388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6344543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6344920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6345109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6345356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6345604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6346010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6346417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6346652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6346884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6347126Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcznh5wyj
2022-12-01T10:45:06.6347401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcznh5wyj/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6347655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_cr66k8
2022-12-01T10:45:06.6347921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_cr66k8/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6348134Z <class 'torch.nn.parameter.Parameter'>
2022-12-01T10:45:06.6348242Z ok (6.702s)
2022-12-01T10:45:06.6348261Z 
2022-12-01T10:45:06.6348577Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6348700Z Ran 1 test in 6.702s
2022-12-01T10:45:06.6348719Z 
2022-12-01T10:45:06.6348794Z OK
2022-12-01T10:45:06.6348831Z 
2022-12-01T10:45:06.6348937Z Generating XML reports...
2022-12-01T10:45:06.6349619Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103140.xml
2022-12-01T10:45:06.6350015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6350193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6350576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6350772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6350797Z 
2022-12-01T10:45:06.6350910Z Running tests...
2022-12-01T10:45:06.6351181Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6351476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6351786Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6352009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29582
2022-12-01T10:45:06.6352227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29583
2022-12-01T10:45:06.6352598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6352777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6353159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6353436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6353810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6353966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6354345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6354537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6354783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6355028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6355430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6355833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6356065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6356296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6356537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fpfejgt
2022-12-01T10:45:06.6356809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fpfejgt/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6357066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi75ols5l
2022-12-01T10:45:06.6357333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi75ols5l/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6357436Z ok (6.994s)
2022-12-01T10:45:06.6357457Z 
2022-12-01T10:45:06.6357727Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6357846Z Ran 1 test in 6.994s
2022-12-01T10:45:06.6357920Z 
2022-12-01T10:45:06.6358020Z OK
2022-12-01T10:45:06.6358039Z 
2022-12-01T10:45:06.6358146Z Generating XML reports...
2022-12-01T10:45:06.6358603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103149.xml
2022-12-01T10:45:06.6358975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6359152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6359530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6359720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6359740Z 
2022-12-01T10:45:06.6359848Z Running tests...
2022-12-01T10:45:06.6360112Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6360418Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6360685Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6360905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29881
2022-12-01T10:45:06.6361124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29882
2022-12-01T10:45:06.6361498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6361676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6362058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6362249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6362688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6362846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6363220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6363408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6363657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6363903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6364304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6364704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6364942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6365173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6365414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv9kp38px
2022-12-01T10:45:06.6365685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv9kp38px/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6365943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc16dhn92
2022-12-01T10:45:06.6366212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc16dhn92/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6366448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6366683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6366786Z ok (7.387s)
2022-12-01T10:45:06.6366809Z 
2022-12-01T10:45:06.6367122Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6367223Z Ran 1 test in 7.388s
2022-12-01T10:45:06.6367261Z 
2022-12-01T10:45:06.6367335Z OK
2022-12-01T10:45:06.6367354Z 
2022-12-01T10:45:06.6367478Z Generating XML reports...
2022-12-01T10:45:06.6367936Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103158.xml
2022-12-01T10:45:06.6368310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6368491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6368873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6369065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6369084Z 
2022-12-01T10:45:06.6369199Z Running tests...
2022-12-01T10:45:06.6369452Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6369764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6370053Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6370274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30234
2022-12-01T10:45:06.6370491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30235
2022-12-01T10:45:06.6370863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6371038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6371418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6371645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6372019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6372191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6372564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6372752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6373000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6373247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6373647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6374047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6374266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6374496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6374753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6fe0ezsc
2022-12-01T10:45:06.6375022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6fe0ezsc/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6375280Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsbuj3vq
2022-12-01T10:45:06.6375547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsbuj3vq/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6376372Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.6376485Z ok (6.455s)
2022-12-01T10:45:06.6376505Z 
2022-12-01T10:45:06.6376773Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6376888Z Ran 1 test in 6.456s
2022-12-01T10:45:06.6376907Z 
2022-12-01T10:45:06.6376982Z OK
2022-12-01T10:45:06.6377020Z 
2022-12-01T10:45:06.6377126Z Generating XML reports...
2022-12-01T10:45:06.6377579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103208.xml
2022-12-01T10:45:06.6377955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6378140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6378527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6378721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6378740Z 
2022-12-01T10:45:06.6378848Z Running tests...
2022-12-01T10:45:06.6379112Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6379466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6379745Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6380493Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.718s)
2022-12-01T10:45:06.6380563Z 
2022-12-01T10:45:06.6380834Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6380948Z Ran 1 test in 1.718s
2022-12-01T10:45:06.6380966Z 
2022-12-01T10:45:06.6381075Z OK (skipped=1)
2022-12-01T10:45:06.6381094Z 
2022-12-01T10:45:06.6381218Z Generating XML reports...
2022-12-01T10:45:06.6381670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103217.xml
2022-12-01T10:45:06.6382043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6382219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6382581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6382777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6382800Z 
2022-12-01T10:45:06.6382908Z Running tests...
2022-12-01T10:45:06.6383168Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6383482Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6383807Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-12-01T10:45:06.6383827Z 
2022-12-01T10:45:06.6384129Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6384244Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6384263Z 
2022-12-01T10:45:06.6384352Z OK (skipped=1)
2022-12-01T10:45:06.6384390Z 
2022-12-01T10:45:06.6384496Z Generating XML reports...
2022-12-01T10:45:06.6384944Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103221.xml
2022-12-01T10:45:06.6385369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6385554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6385936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6386129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6386149Z 
2022-12-01T10:45:06.6386256Z Running tests...
2022-12-01T10:45:06.6386518Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6386814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6387071Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6387819Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.804s)
2022-12-01T10:45:06.6387844Z 
2022-12-01T10:45:06.6388110Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6388222Z Ran 1 test in 1.804s
2022-12-01T10:45:06.6388241Z 
2022-12-01T10:45:06.6388350Z OK (skipped=1)
2022-12-01T10:45:06.6388369Z 
2022-12-01T10:45:06.6388493Z Generating XML reports...
2022-12-01T10:45:06.6388942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103223.xml
2022-12-01T10:45:06.6389316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6389707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6390085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6390365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6390387Z 
2022-12-01T10:45:06.6390495Z Running tests...
2022-12-01T10:45:06.6390761Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6391078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6391350Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6391571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30965
2022-12-01T10:45:06.6391790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30966
2022-12-01T10:45:06.6392142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6392321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6392709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6392902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6393268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6393440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6393813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6394004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6394251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6394479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6394942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6395352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6395586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6395816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6396078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx2pb3y7b
2022-12-01T10:45:06.6396353Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx2pb3y7b/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6396608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn5l5boxp
2022-12-01T10:45:06.6396858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn5l5boxp/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6397655Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T10:45:06.6397993Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T10:45:06.6398782Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T10:45:06.6399117Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T10:45:06.6399271Z ok (7.384s)
2022-12-01T10:45:06.6399290Z 
2022-12-01T10:45:06.6399563Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6399675Z Ran 1 test in 7.384s
2022-12-01T10:45:06.6399695Z 
2022-12-01T10:45:06.6399790Z OK
2022-12-01T10:45:06.6399808Z 
2022-12-01T10:45:06.6399934Z Generating XML reports...
2022-12-01T10:45:06.6400387Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103227.xml
2022-12-01T10:45:06.6400743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6400920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6401301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6401494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6401517Z 
2022-12-01T10:45:06.6401626Z Running tests...
2022-12-01T10:45:06.6401892Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6402208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6402481Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6403232Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.675s)
2022-12-01T10:45:06.6403254Z 
2022-12-01T10:45:06.6403517Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6403613Z Ran 1 test in 1.675s
2022-12-01T10:45:06.6403632Z 
2022-12-01T10:45:06.6403739Z OK (skipped=1)
2022-12-01T10:45:06.6403763Z 
2022-12-01T10:45:06.6403887Z Generating XML reports...
2022-12-01T10:45:06.6404379Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103237.xml
2022-12-01T10:45:06.6404759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6404935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6405317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6405508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6405527Z 
2022-12-01T10:45:06.6405617Z Running tests...
2022-12-01T10:45:06.6405880Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6406198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6406476Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6407220Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.751s)
2022-12-01T10:45:06.6407240Z 
2022-12-01T10:45:06.6407500Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6407612Z Ran 1 test in 1.752s
2022-12-01T10:45:06.6407631Z 
2022-12-01T10:45:06.6407740Z OK (skipped=1)
2022-12-01T10:45:06.6407759Z 
2022-12-01T10:45:06.6407885Z Generating XML reports...
2022-12-01T10:45:06.6408333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103240.xml
2022-12-01T10:45:06.6408685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6408919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6409302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6409496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6409516Z 
2022-12-01T10:45:06.6409624Z Running tests...
2022-12-01T10:45:06.6409887Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6410205Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6410499Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6410701Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31719
2022-12-01T10:45:06.6410922Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31720
2022-12-01T10:45:06.6411303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6411480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6411861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6412052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6412416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6412591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6412967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6413139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6413436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6413691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6414094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6414494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6414725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6414969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6415194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6415434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6415818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6416219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6416478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbw7z28xx
2022-12-01T10:45:06.6416748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbw7z28xx/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6417003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtajrncr
2022-12-01T10:45:06.6417272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtajrncr/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6417510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6417748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6418041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6418255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6418360Z ok (7.589s)
2022-12-01T10:45:06.6418379Z 
2022-12-01T10:45:06.6418650Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6418764Z Ran 1 test in 7.589s
2022-12-01T10:45:06.6418784Z 
2022-12-01T10:45:06.6418880Z OK
2022-12-01T10:45:06.6418899Z 
2022-12-01T10:45:06.6419022Z Generating XML reports...
2022-12-01T10:45:06.6419477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103244.xml
2022-12-01T10:45:06.6419852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6420012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6420407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6420600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6420619Z 
2022-12-01T10:45:06.6420730Z Running tests...
2022-12-01T10:45:06.6421000Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6421316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6421591Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6421811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32132
2022-12-01T10:45:06.6422030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32133
2022-12-01T10:45:06.6422383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6422607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6423000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6423193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6423555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6423730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6424101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6424293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6424522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6424775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6425179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6425575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6425805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6426087Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:45:06.6426317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6426595Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:45:06.6426852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjrq6rtt
2022-12-01T10:45:06.6427178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjrq6rtt/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6427433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdxgfx9zi
2022-12-01T10:45:06.6427705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdxgfx9zi/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6427941Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6428178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6428414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6428649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6428926Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:45:06.6429211Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:45:06.6429693Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:45:06.6429972Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:45:06.6430208Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6430440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6430672Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6430906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6431191Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:45:06.6431533Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:45:06.6431822Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T10:45:06.6432072Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T10:45:06.6432304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6432536Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6432764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6432994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6433097Z ok (8.137s)
2022-12-01T10:45:06.6433121Z 
2022-12-01T10:45:06.6433405Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6433523Z Ran 1 test in 8.137s
2022-12-01T10:45:06.6433543Z 
2022-12-01T10:45:06.6433618Z OK
2022-12-01T10:45:06.6433637Z 
2022-12-01T10:45:06.6433763Z Generating XML reports...
2022-12-01T10:45:06.6434220Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103254.xml
2022-12-01T10:45:06.6434595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6434774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6435155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6435339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6435358Z 
2022-12-01T10:45:06.6435470Z Running tests...
2022-12-01T10:45:06.6435840Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6436163Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6436437Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6437180Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2022-12-01T10:45:06.6437201Z 
2022-12-01T10:45:06.6437468Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6437580Z Ran 1 test in 1.634s
2022-12-01T10:45:06.6437600Z 
2022-12-01T10:45:06.6437708Z OK (skipped=1)
2022-12-01T10:45:06.6437727Z 
2022-12-01T10:45:06.6437833Z Generating XML reports...
2022-12-01T10:45:06.6438288Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103305.xml
2022-12-01T10:45:06.6438661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6438836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6439216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6439408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6439428Z 
2022-12-01T10:45:06.6439535Z Running tests...
2022-12-01T10:45:06.6439797Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6440094Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6440367Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6440642Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32586
2022-12-01T10:45:06.6440868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32587
2022-12-01T10:45:06.6441240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6441415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6441798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6441986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6442349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6442504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6442886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6443077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6443324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6443571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6443968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6444365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6444597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6445148Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:45:06.6445429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6445945Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:45:06.6446202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f9vxvb6
2022-12-01T10:45:06.6446474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f9vxvb6/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6446731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmjd46ayi
2022-12-01T10:45:06.6447007Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmjd46ayi/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6447245Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6447481Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6447756Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T10:45:06.6448031Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T10:45:06.6448310Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T10:45:06.6448606Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T10:45:06.6448984Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T10:45:06.6449321Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T10:45:06.6449648Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T10:45:06.6449961Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T10:45:06.6450196Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6450433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6450538Z ok (7.431s)
2022-12-01T10:45:06.6450558Z 
2022-12-01T10:45:06.6450837Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6450934Z Ran 1 test in 7.431s
2022-12-01T10:45:06.6450954Z 
2022-12-01T10:45:06.6451051Z OK
2022-12-01T10:45:06.6451070Z 
2022-12-01T10:45:06.6451194Z Generating XML reports...
2022-12-01T10:45:06.6451649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103309.xml
2022-12-01T10:45:06.6452021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6452197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6452578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6452771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6452791Z 
2022-12-01T10:45:06.6452989Z Running tests...
2022-12-01T10:45:06.6453242Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6453558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6453949Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6453970Z 
2022-12-01T10:45:06.6454231Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6454341Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6454360Z 
2022-12-01T10:45:06.6454469Z OK (skipped=1)
2022-12-01T10:45:06.6454488Z 
2022-12-01T10:45:06.6454613Z Generating XML reports...
2022-12-01T10:45:06.6455069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103318.xml
2022-12-01T10:45:06.6455442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6455608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6455991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6456184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6456203Z 
2022-12-01T10:45:06.6456314Z Running tests...
2022-12-01T10:45:06.6456577Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6456890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6457278Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6457299Z 
2022-12-01T10:45:06.6457561Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6457727Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6457747Z 
2022-12-01T10:45:06.6457840Z OK (skipped=1)
2022-12-01T10:45:06.6457858Z 
2022-12-01T10:45:06.6457984Z Generating XML reports...
2022-12-01T10:45:06.6458436Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103320.xml
2022-12-01T10:45:06.6458807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6458984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6459367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6459558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6459578Z 
2022-12-01T10:45:06.6459686Z Running tests...
2022-12-01T10:45:06.6459957Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6460258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6460705Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6460725Z 
2022-12-01T10:45:06.6460987Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6461099Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6461119Z 
2022-12-01T10:45:06.6461225Z OK (skipped=1)
2022-12-01T10:45:06.6461244Z 
2022-12-01T10:45:06.6461367Z Generating XML reports...
2022-12-01T10:45:06.6461812Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103322.xml
2022-12-01T10:45:06.6462243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6462421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6462784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6462978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6462998Z 
2022-12-01T10:45:06.6463105Z Running tests...
2022-12-01T10:45:06.6463369Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6463680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6464125Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6464149Z 
2022-12-01T10:45:06.6464416Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6464529Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6464548Z 
2022-12-01T10:45:06.6464656Z OK (skipped=1)
2022-12-01T10:45:06.6464675Z 
2022-12-01T10:45:06.6464781Z Generating XML reports...
2022-12-01T10:45:06.6465227Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103325.xml
2022-12-01T10:45:06.6465598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6465776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6466155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6466348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6466371Z 
2022-12-01T10:45:06.6466481Z Running tests...
2022-12-01T10:45:06.6466790Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6467116Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6467545Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6467584Z 
2022-12-01T10:45:06.6467827Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6467939Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6467959Z 
2022-12-01T10:45:06.6468065Z OK (skipped=1)
2022-12-01T10:45:06.6468084Z 
2022-12-01T10:45:06.6468207Z Generating XML reports...
2022-12-01T10:45:06.6468656Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103327.xml
2022-12-01T10:45:06.6469035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6469212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6469820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6469999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6470038Z 
2022-12-01T10:45:06.6470130Z Running tests...
2022-12-01T10:45:06.6470397Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6470713Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6471158Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6471252Z 
2022-12-01T10:45:06.6471529Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6471645Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6471664Z 
2022-12-01T10:45:06.6471773Z OK (skipped=1)
2022-12-01T10:45:06.6471792Z 
2022-12-01T10:45:06.6471918Z Generating XML reports...
2022-12-01T10:45:06.6472365Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103329.xml
2022-12-01T10:45:06.6472720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6472898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6473281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6473475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6473498Z 
2022-12-01T10:45:06.6473605Z Running tests...
2022-12-01T10:45:06.6473871Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6474186Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6474633Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6474653Z 
2022-12-01T10:45:06.6474916Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6475012Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6475032Z 
2022-12-01T10:45:06.6475139Z OK (skipped=1)
2022-12-01T10:45:06.6475158Z 
2022-12-01T10:45:06.6475282Z Generating XML reports...
2022-12-01T10:45:06.6475731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103331.xml
2022-12-01T10:45:06.6476162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6476345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6476729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6476922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6476942Z 
2022-12-01T10:45:06.6477050Z Running tests...
2022-12-01T10:45:06.6477293Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6477607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6478050Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6478077Z 
2022-12-01T10:45:06.6478340Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6478454Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6478474Z 
2022-12-01T10:45:06.6478583Z OK (skipped=1)
2022-12-01T10:45:06.6478602Z 
2022-12-01T10:45:06.6478726Z Generating XML reports...
2022-12-01T10:45:06.6479177Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103333.xml
2022-12-01T10:45:06.6479597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6479756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6480139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6480331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6480398Z 
2022-12-01T10:45:06.6480514Z Running tests...
2022-12-01T10:45:06.6480780Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6481090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6481525Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6481546Z 
2022-12-01T10:45:06.6481811Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6481923Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6481943Z 
2022-12-01T10:45:06.6482033Z OK (skipped=1)
2022-12-01T10:45:06.6482070Z 
2022-12-01T10:45:06.6482176Z Generating XML reports...
2022-12-01T10:45:06.6482625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103335.xml
2022-12-01T10:45:06.6483004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6483178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6483554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6483744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6483763Z 
2022-12-01T10:45:06.6483870Z Running tests...
2022-12-01T10:45:06.6484134Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6484431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6484922Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6484947Z 
2022-12-01T10:45:06.6485217Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6485330Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6485350Z 
2022-12-01T10:45:06.6485460Z OK (skipped=1)
2022-12-01T10:45:06.6485479Z 
2022-12-01T10:45:06.6485601Z Generating XML reports...
2022-12-01T10:45:06.6486048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103337.xml
2022-12-01T10:45:06.6486420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6486598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6486962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6487159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6487182Z 
2022-12-01T10:45:06.6487289Z Running tests...
2022-12-01T10:45:06.6487553Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6487868Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6488254Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6488275Z 
2022-12-01T10:45:06.6488536Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6488647Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6488666Z 
2022-12-01T10:45:06.6488774Z OK (skipped=1)
2022-12-01T10:45:06.6488792Z 
2022-12-01T10:45:06.6488898Z Generating XML reports...
2022-12-01T10:45:06.6489344Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103339.xml
2022-12-01T10:45:06.6489788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6489964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6490341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6490530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6490550Z 
2022-12-01T10:45:06.6490659Z Running tests...
2022-12-01T10:45:06.6490919Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6491233Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6491603Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T10:45:06.6491646Z 
2022-12-01T10:45:06.6491894Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6492007Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6492027Z 
2022-12-01T10:45:06.6492134Z OK (skipped=1)
2022-12-01T10:45:06.6492153Z 
2022-12-01T10:45:06.6492275Z Generating XML reports...
2022-12-01T10:45:06.6492719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103341.xml
2022-12-01T10:45:06.6493087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6493265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6493648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6493826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6493865Z 
2022-12-01T10:45:06.6494001Z Running tests...
2022-12-01T10:45:06.6494278Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6494590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6494856Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6495601Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.744s)
2022-12-01T10:45:06.6495622Z 
2022-12-01T10:45:06.6495884Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6495998Z Ran 1 test in 1.745s
2022-12-01T10:45:06.6496022Z 
2022-12-01T10:45:06.6496131Z OK (skipped=1)
2022-12-01T10:45:06.6496150Z 
2022-12-01T10:45:06.6496278Z Generating XML reports...
2022-12-01T10:45:06.6496711Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103343.xml
2022-12-01T10:45:06.6497084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6497257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6497636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6497829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6497849Z 
2022-12-01T10:45:06.6497957Z Running tests...
2022-12-01T10:45:06.6498219Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6498535Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6498839Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6499062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34071
2022-12-01T10:45:06.6499281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34072
2022-12-01T10:45:06.6499654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6499828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6500208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6500401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6500763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6500942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6501303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6501491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6501739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6501983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6502383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6502779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6503011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6503285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6503552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9kmwvqjs
2022-12-01T10:45:06.6503808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9kmwvqjs/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6504062Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps06a_dkj
2022-12-01T10:45:06.6504329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps06a_dkj/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6504432Z ok (6.580s)
2022-12-01T10:45:06.6504452Z 
2022-12-01T10:45:06.6504720Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6504833Z Ran 1 test in 6.580s
2022-12-01T10:45:06.6504852Z 
2022-12-01T10:45:06.6504946Z OK
2022-12-01T10:45:06.6504965Z 
2022-12-01T10:45:06.6505089Z Generating XML reports...
2022-12-01T10:45:06.6505528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103347.xml
2022-12-01T10:45:06.6505903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6506078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6506455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6506645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6506665Z 
2022-12-01T10:45:06.6506772Z Running tests...
2022-12-01T10:45:06.6507032Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6507353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6507608Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6507886Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34463
2022-12-01T10:45:06.6508103Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34464
2022-12-01T10:45:06.6508474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6508651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6509029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6509220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6509809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6509985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6510354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6510548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6510795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6511037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6511441Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6511838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6512066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6512296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6512632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj439srz
2022-12-01T10:45:06.6512894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj439srz/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6513147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtneal0n
2022-12-01T10:45:06.6513415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtneal0n/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6513649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6513883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6514290Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpxtneal0n'>
2022-12-01T10:45:06.6514458Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:45:06.6514853Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpnj439srz'>
2022-12-01T10:45:06.6515005Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:45:06.6515108Z ok (7.488s)
2022-12-01T10:45:06.6515129Z 
2022-12-01T10:45:06.6515398Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6515513Z Ran 1 test in 7.488s
2022-12-01T10:45:06.6515532Z 
2022-12-01T10:45:06.6515624Z OK
2022-12-01T10:45:06.6515643Z 
2022-12-01T10:45:06.6515768Z Generating XML reports...
2022-12-01T10:45:06.6516221Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103356.xml
2022-12-01T10:45:06.6516592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6516767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6517127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6517387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6517406Z 
2022-12-01T10:45:06.6517513Z Running tests...
2022-12-01T10:45:06.6517777Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6518091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6518366Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-12-01T10:45:06.6518386Z 
2022-12-01T10:45:06.6518648Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6518761Z Ran 1 test in 0.006s
2022-12-01T10:45:06.6518780Z 
2022-12-01T10:45:06.6518869Z OK (skipped=1)
2022-12-01T10:45:06.6518906Z 
2022-12-01T10:45:06.6519012Z Generating XML reports...
2022-12-01T10:45:06.6519462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103406.xml
2022-12-01T10:45:06.6519842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6520020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6520398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6520587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6520607Z 
2022-12-01T10:45:06.6520717Z Running tests...
2022-12-01T10:45:06.6520981Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6521277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6521538Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6521762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34936
2022-12-01T10:45:06.6522026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34937
2022-12-01T10:45:06.6522408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6522587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6522968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6523160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6523525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6523683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6524060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6524255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6524502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6524749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6525150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6525548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6525780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6525993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6526249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc12r6_hg
2022-12-01T10:45:06.6526576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc12r6_hg/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6526831Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8vnngk3a
2022-12-01T10:45:06.6527102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8vnngk3a/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6527337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6527575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6527678Z ok (7.376s)
2022-12-01T10:45:06.6527698Z 
2022-12-01T10:45:06.6527968Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6528065Z Ran 1 test in 7.376s
2022-12-01T10:45:06.6528084Z 
2022-12-01T10:45:06.6528176Z OK
2022-12-01T10:45:06.6528195Z 
2022-12-01T10:45:06.6528320Z Generating XML reports...
2022-12-01T10:45:06.6528782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103408.xml
2022-12-01T10:45:06.6529154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6529331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6529713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6529903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6529923Z 
2022-12-01T10:45:06.6530031Z Running tests...
2022-12-01T10:45:06.6530274Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6530586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6530871Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6531135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35369
2022-12-01T10:45:06.6531361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35370
2022-12-01T10:45:06.6531734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6531914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6532293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6532467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6532832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6533005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6533394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6533580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6533825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6534067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6534471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6534866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6535078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6535308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6535604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6535844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6536242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6536636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6536876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.6537116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.6537506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6537888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6538147Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jutl6o6
2022-12-01T10:45:06.6538421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jutl6o6/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6538677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgllp2fe5
2022-12-01T10:45:06.6538944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgllp2fe5/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6539047Z ok (5.758s)
2022-12-01T10:45:06.6539068Z 
2022-12-01T10:45:06.6539339Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6539456Z Ran 1 test in 5.758s
2022-12-01T10:45:06.6539475Z 
2022-12-01T10:45:06.6539550Z OK
2022-12-01T10:45:06.6539587Z 
2022-12-01T10:45:06.6539691Z Generating XML reports...
2022-12-01T10:45:06.6540193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103418.xml
2022-12-01T10:45:06.6540577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6540753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6541133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6541323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6541343Z 
2022-12-01T10:45:06.6541452Z Running tests...
2022-12-01T10:45:06.6541718Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6542015Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6542300Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6542531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35768
2022-12-01T10:45:06.6542751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35769
2022-12-01T10:45:06.6543120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6543296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6543675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6543867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6544211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6544383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6544753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6544999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6545245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6545488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6545887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6546281Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6546512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6546723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6546967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6547360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6547601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.6547844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6548237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6548475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.6548866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6549253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6549800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1blct9x_
2022-12-01T10:45:06.6550074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1blct9x_/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6550328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxnfv5qi
2022-12-01T10:45:06.6550593Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxnfv5qi/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6551060Z [W ProcessGroupNCCL.cpp:948] [Rank 0] Found key in store: NCCLABORTEDCOMM:20a939ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-12-01T10:45:06.6551518Z [W ProcessGroupNCCL.cpp:948] [Rank 1] Found key in store: NCCLABORTEDCOMM:20a939ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2022-12-01T10:45:06.6551627Z ok (16.263s)
2022-12-01T10:45:06.6551647Z 
2022-12-01T10:45:06.6551927Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6552044Z Ran 1 test in 16.264s
2022-12-01T10:45:06.6552064Z 
2022-12-01T10:45:06.6552161Z OK
2022-12-01T10:45:06.6552180Z 
2022-12-01T10:45:06.6552304Z Generating XML reports...
2022-12-01T10:45:06.6552741Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103426.xml
2022-12-01T10:45:06.6553115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6553371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6553758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6553952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6553972Z 
2022-12-01T10:45:06.6554080Z Running tests...
2022-12-01T10:45:06.6554344Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6554661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6554970Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6555173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36171
2022-12-01T10:45:06.6555391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36172
2022-12-01T10:45:06.6555777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6555953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6556337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6556527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6556893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6557069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6557423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6557616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6557864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6558155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6558564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6558961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6559191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6559422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6559682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoz7n5pf6
2022-12-01T10:45:06.6559935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoz7n5pf6/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6560194Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymdego81
2022-12-01T10:45:06.6560464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymdego81/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6560567Z ok (7.907s)
2022-12-01T10:45:06.6560586Z 
2022-12-01T10:45:06.6560856Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6560970Z Ran 1 test in 7.907s
2022-12-01T10:45:06.6560990Z 
2022-12-01T10:45:06.6561085Z OK
2022-12-01T10:45:06.6561104Z 
2022-12-01T10:45:06.6561228Z Generating XML reports...
2022-12-01T10:45:06.6561680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103444.xml
2022-12-01T10:45:06.6562035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6562214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6562656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6562848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6562868Z 
2022-12-01T10:45:06.6562978Z Running tests...
2022-12-01T10:45:06.6563249Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6563566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6563855Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6564056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36604
2022-12-01T10:45:06.6564278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36605
2022-12-01T10:45:06.6564650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6564831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6565213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6565403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6565765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6565939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6566319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6566489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6566738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6566986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6567436Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6567843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6568074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6568303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6568565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8dxcy9_
2022-12-01T10:45:06.6568839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8dxcy9_/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6569078Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpft27at54
2022-12-01T10:45:06.6569345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpft27at54/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6569452Z ok (7.853s)
2022-12-01T10:45:06.6569471Z 
2022-12-01T10:45:06.6569742Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6569856Z Ran 1 test in 7.853s
2022-12-01T10:45:06.6569875Z 
2022-12-01T10:45:06.6569972Z OK
2022-12-01T10:45:06.6569991Z 
2022-12-01T10:45:06.6570115Z Generating XML reports...
2022-12-01T10:45:06.6570568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103454.xml
2022-12-01T10:45:06.6570921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6571100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6571484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6571726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6571749Z 
2022-12-01T10:45:06.6571858Z Running tests...
2022-12-01T10:45:06.6572121Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6572435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6572692Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6572894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37106
2022-12-01T10:45:06.6573115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37107
2022-12-01T10:45:06.6573486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6573662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6574045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6574244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6574612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6574787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6575167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6575339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6575586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6575827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6576225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6576672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6576909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6577137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6577396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e_qo9xf
2022-12-01T10:45:06.6577666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e_qo9xf/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6577899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkhn6scvx
2022-12-01T10:45:06.6578163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkhn6scvx/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6578266Z ok (7.486s)
2022-12-01T10:45:06.6578289Z 
2022-12-01T10:45:06.6578558Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6578676Z Ran 1 test in 7.487s
2022-12-01T10:45:06.6578696Z 
2022-12-01T10:45:06.6578788Z OK
2022-12-01T10:45:06.6578807Z 
2022-12-01T10:45:06.6578931Z Generating XML reports...
2022-12-01T10:45:06.6579431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103505.xml
2022-12-01T10:45:06.6579788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6579965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6580345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6580537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6580557Z 
2022-12-01T10:45:06.6580667Z Running tests...
2022-12-01T10:45:06.6580989Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6581310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6581571Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6581792Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37506
2022-12-01T10:45:06.6581991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37507
2022-12-01T10:45:06.6582364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6582542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6582920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6583115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6583485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6583660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6584037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6584207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6584452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6584697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6585100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6585497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6585773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6586006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6586267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntuww58j
2022-12-01T10:45:06.6586539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntuww58j/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6586773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_2gl7bym
2022-12-01T10:45:06.6587040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_2gl7bym/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6587821Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.6588604Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.6588848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6589132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6589234Z ok (7.337s)
2022-12-01T10:45:06.6589254Z 
2022-12-01T10:45:06.6589746Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6589864Z Ran 1 test in 7.337s
2022-12-01T10:45:06.6589884Z 
2022-12-01T10:45:06.6589982Z OK
2022-12-01T10:45:06.6590001Z 
2022-12-01T10:45:06.6590125Z Generating XML reports...
2022-12-01T10:45:06.6590586Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103514.xml
2022-12-01T10:45:06.6590941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6591122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6591503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6591698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6591721Z 
2022-12-01T10:45:06.6591830Z Running tests...
2022-12-01T10:45:06.6592095Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6592408Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6592690Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6593536Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.793s)
2022-12-01T10:45:06.6593558Z 
2022-12-01T10:45:06.6593865Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6593968Z Ran 1 test in 1.793s
2022-12-01T10:45:06.6593987Z 
2022-12-01T10:45:06.6594214Z OK (skipped=1)
2022-12-01T10:45:06.6594236Z 
2022-12-01T10:45:06.6594384Z Generating XML reports...
2022-12-01T10:45:06.6595023Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103524.xml
2022-12-01T10:45:06.6595440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6595650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6596067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6596296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6596316Z 
2022-12-01T10:45:06.6596459Z Running tests...
2022-12-01T10:45:06.6596710Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6597082Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6597401Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6598227Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.725s)
2022-12-01T10:45:06.6598247Z 
2022-12-01T10:45:06.6598547Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6598695Z Ran 1 test in 1.726s
2022-12-01T10:45:06.6598714Z 
2022-12-01T10:45:06.6598855Z OK (skipped=1)
2022-12-01T10:45:06.6598874Z 
2022-12-01T10:45:06.6599053Z Generating XML reports...
2022-12-01T10:45:06.6599557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103528.xml
2022-12-01T10:45:06.6600053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6600214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6600629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6600901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6600921Z 
2022-12-01T10:45:06.6601066Z Running tests...
2022-12-01T10:45:06.6601367Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6601726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6602039Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6602299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38214
2022-12-01T10:45:06.6602503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38215
2022-12-01T10:45:06.6602916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6603127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6603578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6603805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6604213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6604453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6604865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6605140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6605377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6605658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6606097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6606567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6606845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6607111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6607402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rgq93qn
2022-12-01T10:45:06.6607712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rgq93qn/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6608003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr30i2gno
2022-12-01T10:45:06.6608254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr30i2gno/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6608635Z STAGE:2022-12-01 10:35:38 38215:38215 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6609007Z STAGE:2022-12-01 10:35:38 38214:38214 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6609322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6609594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6610002Z STAGE:2022-12-01 10:35:38 38215:38215 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6622107Z STAGE:2022-12-01 10:35:38 38214:38214 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6622947Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.6623732Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.6624327Z STAGE:2022-12-01 10:35:38 38214:38214 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:35:38 38215:38215 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6624351Z 
2022-12-01T10:45:06.6624694Z STAGE:2022-12-01 10:35:38 38215:38215 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6625028Z STAGE:2022-12-01 10:35:38 38214:38214 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6625134Z ok (6.740s)
2022-12-01T10:45:06.6625154Z 
2022-12-01T10:45:06.6625426Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6625524Z Ran 1 test in 6.741s
2022-12-01T10:45:06.6625547Z 
2022-12-01T10:45:06.6625643Z OK
2022-12-01T10:45:06.6625662Z 
2022-12-01T10:45:06.6625788Z Generating XML reports...
2022-12-01T10:45:06.6626375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103532.xml
2022-12-01T10:45:06.6626774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6626952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6627337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6627531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6627551Z 
2022-12-01T10:45:06.6627644Z Running tests...
2022-12-01T10:45:06.6627915Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6628229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6628506Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6628729Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38607
2022-12-01T10:45:06.6628947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38608
2022-12-01T10:45:06.6629321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6629825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6630226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6630400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6630767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6631049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6631431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6631619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6631867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6632111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6632515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6632897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6633132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6633371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6633629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxnnymll1
2022-12-01T10:45:06.6633903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxnnymll1/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6634159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqqd4ndw
2022-12-01T10:45:06.6634429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqqd4ndw/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6634535Z ok (6.942s)
2022-12-01T10:45:06.6634556Z 
2022-12-01T10:45:06.6634825Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6634922Z Ran 1 test in 6.942s
2022-12-01T10:45:06.6634942Z 
2022-12-01T10:45:06.6635037Z OK
2022-12-01T10:45:06.6635057Z 
2022-12-01T10:45:06.6635182Z Generating XML reports...
2022-12-01T10:45:06.6635704Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103541.xml
2022-12-01T10:45:06.6636094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6636271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6636653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6636845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6636865Z 
2022-12-01T10:45:06.6636974Z Running tests...
2022-12-01T10:45:06.6637222Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6637538Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6637818Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6638581Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.730s)
2022-12-01T10:45:06.6638603Z 
2022-12-01T10:45:06.6638867Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6638980Z Ran 1 test in 1.730s
2022-12-01T10:45:06.6638999Z 
2022-12-01T10:45:06.6639108Z OK (skipped=1)
2022-12-01T10:45:06.6639127Z 
2022-12-01T10:45:06.6639255Z Generating XML reports...
2022-12-01T10:45:06.6639706Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103550.xml
2022-12-01T10:45:06.6640062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6640293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6640681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6640874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6640893Z 
2022-12-01T10:45:06.6641004Z Running tests...
2022-12-01T10:45:06.6641268Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6641583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6641866Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6642087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39165
2022-12-01T10:45:06.6642290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39166
2022-12-01T10:45:06.6642665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6642852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6643237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6643430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6643795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6643975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6644362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6644534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6644782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6645082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6645500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6645897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6646129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6646359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6646624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpczukp8ah
2022-12-01T10:45:06.6646900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpczukp8ah/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6647137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvuxo6y_6
2022-12-01T10:45:06.6647419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvuxo6y_6/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6648336Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:45:06.6648452Z   warnings.warn(
2022-12-01T10:45:06.6649360Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:45:06.6649520Z   warnings.warn(
2022-12-01T10:45:06.6649766Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6649998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6650101Z ok (6.007s)
2022-12-01T10:45:06.6650120Z 
2022-12-01T10:45:06.6650388Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6650484Z Ran 1 test in 6.008s
2022-12-01T10:45:06.6650522Z 
2022-12-01T10:45:06.6650599Z OK
2022-12-01T10:45:06.6650617Z 
2022-12-01T10:45:06.6650745Z Generating XML reports...
2022-12-01T10:45:06.6651200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103554.xml
2022-12-01T10:45:06.6651576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6651759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6652144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6652336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6652356Z 
2022-12-01T10:45:06.6652465Z Running tests...
2022-12-01T10:45:06.6652713Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6653032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6653314Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6654062Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s)
2022-12-01T10:45:06.6654086Z 
2022-12-01T10:45:06.6654389Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6654509Z Ran 1 test in 1.658s
2022-12-01T10:45:06.6654528Z 
2022-12-01T10:45:06.6654638Z OK (skipped=1)
2022-12-01T10:45:06.6654656Z 
2022-12-01T10:45:06.6654782Z Generating XML reports...
2022-12-01T10:45:06.6655239Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103603.xml
2022-12-01T10:45:06.6655611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6655771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6656151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6656345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6656367Z 
2022-12-01T10:45:06.6656477Z Running tests...
2022-12-01T10:45:06.6656745Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6657059Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6657335Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6657559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39628
2022-12-01T10:45:06.6657760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39629
2022-12-01T10:45:06.6658137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6658313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6658696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6658958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6659323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6659497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6659876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6660069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6660298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6660544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6660946Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6661354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6661586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6661817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6662077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmckrtw1
2022-12-01T10:45:06.6662351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmckrtw1/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6662588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6kp64kb7
2022-12-01T10:45:06.6662859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6kp64kb7/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6663206Z STAGE:2022-12-01 10:36:12 39629:39629 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6663886Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.6664007Z   warnings.warn(
2022-12-01T10:45:06.6664343Z STAGE:2022-12-01 10:36:12 39628:39628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6664961Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T10:45:06.6665073Z   warnings.warn(
2022-12-01T10:45:06.6665313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6665549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:45:06.6665884Z STAGE:2022-12-01 10:36:12 39629:39629 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6666218Z STAGE:2022-12-01 10:36:12 39628:39628 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6666545Z STAGE:2022-12-01 10:36:12 39628:39628 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6666880Z STAGE:2022-12-01 10:36:13 39628:39628 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6666983Z ok (7.151s)
2022-12-01T10:45:06.6667003Z 
2022-12-01T10:45:06.6667272Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6667390Z Ran 1 test in 7.152s
2022-12-01T10:45:06.6667410Z 
2022-12-01T10:45:06.6667505Z OK
2022-12-01T10:45:06.6667524Z 
2022-12-01T10:45:06.6667630Z Generating XML reports...
2022-12-01T10:45:06.6668084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103606.xml
2022-12-01T10:45:06.6668520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6668698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6669082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6669277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6669297Z 
2022-12-01T10:45:06.6669644Z Running tests...
2022-12-01T10:45:06.6669926Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6670245Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6670492Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6670716Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40031
2022-12-01T10:45:06.6670946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40032
2022-12-01T10:45:06.6671319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6671497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6671877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6672068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6672429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6672586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6672959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6673153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6673479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6673737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6674142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6674541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6674774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6675003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6675240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgtywtcw
2022-12-01T10:45:06.6675523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgtywtcw/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6675780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvi18u1w
2022-12-01T10:45:06.6676051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvi18u1w/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6676154Z ok (6.864s)
2022-12-01T10:45:06.6676174Z 
2022-12-01T10:45:06.6676443Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6676556Z Ran 1 test in 6.864s
2022-12-01T10:45:06.6676576Z 
2022-12-01T10:45:06.6676670Z OK
2022-12-01T10:45:06.6676689Z 
2022-12-01T10:45:06.6676793Z Generating XML reports...
2022-12-01T10:45:06.6677248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103616.xml
2022-12-01T10:45:06.6677619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6677870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6678254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6678447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6678466Z 
2022-12-01T10:45:06.6678576Z Running tests...
2022-12-01T10:45:06.6678841Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6679156Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6679464Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6679689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40525
2022-12-01T10:45:06.6679908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40526
2022-12-01T10:45:06.6680289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6680469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6680845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6681037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6681399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6681556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6681927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6682115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6682365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6682661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6683072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6683469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6683701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6683930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6684223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpklm_no_v
2022-12-01T10:45:06.6684497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpklm_no_v/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6684755Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbj0qct6m
2022-12-01T10:45:06.6685029Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbj0qct6m/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6685133Z ok (6.915s)
2022-12-01T10:45:06.6685152Z 
2022-12-01T10:45:06.6685425Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6685540Z Ran 1 test in 6.915s
2022-12-01T10:45:06.6685560Z 
2022-12-01T10:45:06.6685655Z OK
2022-12-01T10:45:06.6685673Z 
2022-12-01T10:45:06.6685797Z Generating XML reports...
2022-12-01T10:45:06.6686231Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103625.xml
2022-12-01T10:45:06.6686607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6686785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6687231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6687424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6687444Z 
2022-12-01T10:45:06.6687555Z Running tests...
2022-12-01T10:45:06.6687822Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6688138Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6688395Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6689142Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s)
2022-12-01T10:45:06.6689186Z 
2022-12-01T10:45:06.6689430Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6689546Z Ran 1 test in 1.657s
2022-12-01T10:45:06.6689566Z 
2022-12-01T10:45:06.6689673Z OK (skipped=1)
2022-12-01T10:45:06.6689692Z 
2022-12-01T10:45:06.6689817Z Generating XML reports...
2022-12-01T10:45:06.6690269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103634.xml
2022-12-01T10:45:06.6690643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6690820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6691202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6691374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6691414Z 
2022-12-01T10:45:06.6691504Z Running tests...
2022-12-01T10:45:06.6691774Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6692133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6692401Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6693150Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2022-12-01T10:45:06.6693171Z 
2022-12-01T10:45:06.6693438Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6693552Z Ran 1 test in 1.666s
2022-12-01T10:45:06.6693572Z 
2022-12-01T10:45:06.6693682Z OK (skipped=1)
2022-12-01T10:45:06.6693701Z 
2022-12-01T10:45:06.6693825Z Generating XML reports...
2022-12-01T10:45:06.6694269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103638.xml
2022-12-01T10:45:06.6694642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6694818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6695200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6695392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6695411Z 
2022-12-01T10:45:06.6695522Z Running tests...
2022-12-01T10:45:06.6695788Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6696107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6696381Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6697179Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.740s)
2022-12-01T10:45:06.6697218Z 
2022-12-01T10:45:06.6697465Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6697576Z Ran 1 test in 1.740s
2022-12-01T10:45:06.6697595Z 
2022-12-01T10:45:06.6697703Z OK (skipped=1)
2022-12-01T10:45:06.6697722Z 
2022-12-01T10:45:06.6697847Z Generating XML reports...
2022-12-01T10:45:06.6698297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103642.xml
2022-12-01T10:45:06.6698671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6698854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6699236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6699408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6699427Z 
2022-12-01T10:45:06.6699536Z Running tests...
2022-12-01T10:45:06.6699800Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6700114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6700410Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6700630Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41258
2022-12-01T10:45:06.6700853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41259
2022-12-01T10:45:06.6701274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6701440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6701820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6702012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6702379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6702558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6702939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6703128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6703384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6703633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6704018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6704420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6704655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6704888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6705146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nkyymjg
2022-12-01T10:45:06.6705420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nkyymjg/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6705735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv768a20q
2022-12-01T10:45:06.6705999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv768a20q/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6706102Z ok (7.308s)
2022-12-01T10:45:06.6706122Z 
2022-12-01T10:45:06.6706372Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6706485Z Ran 1 test in 7.308s
2022-12-01T10:45:06.6706504Z 
2022-12-01T10:45:06.6706599Z OK
2022-12-01T10:45:06.6706618Z 
2022-12-01T10:45:06.6706743Z Generating XML reports...
2022-12-01T10:45:06.6707197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103646.xml
2022-12-01T10:45:06.6707572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6707751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6708140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6708312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6708351Z 
2022-12-01T10:45:06.6708442Z Running tests...
2022-12-01T10:45:06.6708707Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6709026Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6709288Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6709748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41525
2022-12-01T10:45:06.6709977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41526
2022-12-01T10:45:06.6710357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6710541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6710976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6711180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6711549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6711725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6712105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6712297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6712547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6712801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6713190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6713590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6713822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6714067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6714291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6714531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6714926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6715398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6715506Z ok (4.122s)
2022-12-01T10:45:06.6715525Z 
2022-12-01T10:45:06.6715774Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6715889Z Ran 1 test in 4.122s
2022-12-01T10:45:06.6715908Z 
2022-12-01T10:45:06.6716002Z OK
2022-12-01T10:45:06.6716021Z 
2022-12-01T10:45:06.6716145Z Generating XML reports...
2022-12-01T10:45:06.6716598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103655.xml
2022-12-01T10:45:06.6716971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6717148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6717529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6717733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6717753Z 
2022-12-01T10:45:06.6717843Z Running tests...
2022-12-01T10:45:06.6718109Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6718425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6718682Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6718904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41795
2022-12-01T10:45:06.6719123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41796
2022-12-01T10:45:06.6719497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6719677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6720092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6720296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6720666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6720842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6721224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6721419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6721664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6721910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6722322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6722702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6722933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6723177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6723402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6723640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6724037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6724431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6724597Z ok (4.322s)
2022-12-01T10:45:06.6724619Z 
2022-12-01T10:45:06.6724888Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6724985Z Ran 1 test in 4.322s
2022-12-01T10:45:06.6725004Z 
2022-12-01T10:45:06.6725097Z OK
2022-12-01T10:45:06.6725116Z 
2022-12-01T10:45:06.6725242Z Generating XML reports...
2022-12-01T10:45:06.6725696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103702.xml
2022-12-01T10:45:06.6726068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6726246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6726627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6726820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6726844Z 
2022-12-01T10:45:06.6726957Z Running tests...
2022-12-01T10:45:06.6727205Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6727521Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6727799Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6728548Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.765s)
2022-12-01T10:45:06.6728569Z 
2022-12-01T10:45:06.6728832Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6728945Z Ran 1 test in 1.765s
2022-12-01T10:45:06.6728968Z 
2022-12-01T10:45:06.6729078Z OK (skipped=1)
2022-12-01T10:45:06.6729096Z 
2022-12-01T10:45:06.6729265Z Generating XML reports...
2022-12-01T10:45:06.6729727Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103708.xml
2022-12-01T10:45:06.6730084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6730263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6730646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6730839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6730858Z 
2022-12-01T10:45:06.6730967Z Running tests...
2022-12-01T10:45:06.6731231Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6731547Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6731834Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6732581Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.746s)
2022-12-01T10:45:06.6732602Z 
2022-12-01T10:45:06.6732867Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6732962Z Ran 1 test in 1.747s
2022-12-01T10:45:06.6732981Z 
2022-12-01T10:45:06.6733091Z OK (skipped=1)
2022-12-01T10:45:06.6733110Z 
2022-12-01T10:45:06.6733237Z Generating XML reports...
2022-12-01T10:45:06.6733689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103712.xml
2022-12-01T10:45:06.6734129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6734306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6734686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6734879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6734899Z 
2022-12-01T10:45:06.6735010Z Running tests...
2022-12-01T10:45:06.6735260Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6735574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6735848Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6736069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42358
2022-12-01T10:45:06.6736298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42359
2022-12-01T10:45:06.6736671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6736846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6737229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6737405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6737774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6737954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6738330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6738522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6738813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6739069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6739472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6739872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6740087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6740319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6740425Z ok (4.061s)
2022-12-01T10:45:06.6740444Z 
2022-12-01T10:45:06.6740715Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6740834Z Ran 1 test in 4.061s
2022-12-01T10:45:06.6740856Z 
2022-12-01T10:45:06.6740951Z OK
2022-12-01T10:45:06.6740970Z 
2022-12-01T10:45:06.6741097Z Generating XML reports...
2022-12-01T10:45:06.6741550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103716.xml
2022-12-01T10:45:06.6741901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6742081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6742464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6742658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6742677Z 
2022-12-01T10:45:06.6742789Z Running tests...
2022-12-01T10:45:06.6743054Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6743425Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6743677Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.6743697Z 
2022-12-01T10:45:06.6743958Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6744053Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6744072Z 
2022-12-01T10:45:06.6744182Z OK (skipped=1)
2022-12-01T10:45:06.6744202Z 
2022-12-01T10:45:06.6744326Z Generating XML reports...
2022-12-01T10:45:06.6744775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103723.xml
2022-12-01T10:45:06.6745146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6745324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6745711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6745904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6745923Z 
2022-12-01T10:45:06.6746031Z Running tests...
2022-12-01T10:45:06.6746277Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6746591Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6746855Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.004s)
2022-12-01T10:45:06.6746875Z 
2022-12-01T10:45:06.6747142Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6747256Z Ran 1 test in 0.004s
2022-12-01T10:45:06.6747276Z 
2022-12-01T10:45:06.6747386Z OK (skipped=1)
2022-12-01T10:45:06.6747405Z 
2022-12-01T10:45:06.6747531Z Generating XML reports...
2022-12-01T10:45:06.6748027Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103725.xml
2022-12-01T10:45:06.6748391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6748568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6748948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6749142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6749161Z 
2022-12-01T10:45:06.6749271Z Running tests...
2022-12-01T10:45:06.6749765Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6750089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6750339Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6750571Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42787
2022-12-01T10:45:06.6750774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42788
2022-12-01T10:45:06.6751147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6751325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6751704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6751897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6752260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6752435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6752816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6753074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6753324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6753570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6753975Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6754373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6754607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6754839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6755184Z STAGE:2022-12-01 10:37:32 42788:42788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6755513Z STAGE:2022-12-01 10:37:33 42787:42787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6755832Z STAGE:2022-12-01 10:37:34 42787:42787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6756168Z STAGE:2022-12-01 10:37:34 42788:42788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6756500Z STAGE:2022-12-01 10:37:34 42787:42787 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6756826Z STAGE:2022-12-01 10:37:34 42788:42788 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.6757155Z STAGE:2022-12-01 10:37:34 42788:42788 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6757481Z STAGE:2022-12-01 10:37:34 42787:42787 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.6757589Z ok (7.891s)
2022-12-01T10:45:06.6757609Z 
2022-12-01T10:45:06.6757971Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6758093Z Ran 1 test in 7.891s
2022-12-01T10:45:06.6758113Z 
2022-12-01T10:45:06.6758187Z OK
2022-12-01T10:45:06.6758205Z 
2022-12-01T10:45:06.6758331Z Generating XML reports...
2022-12-01T10:45:06.6758785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103727.xml
2022-12-01T10:45:06.6759157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6759333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6759714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6759907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6759930Z 
2022-12-01T10:45:06.6760041Z Running tests...
2022-12-01T10:45:06.6760291Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6760614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6760883Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.6760903Z 
2022-12-01T10:45:06.6761169Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6761283Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6761303Z 
2022-12-01T10:45:06.6761416Z OK (skipped=1)
2022-12-01T10:45:06.6761435Z 
2022-12-01T10:45:06.6761564Z Generating XML reports...
2022-12-01T10:45:06.6762017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103737.xml
2022-12-01T10:45:06.6762391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6762608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6762994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6763184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6763204Z 
2022-12-01T10:45:06.6763312Z Running tests...
2022-12-01T10:45:06.6763576Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6763893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6764155Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.6764175Z 
2022-12-01T10:45:06.6764439Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6764553Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6764577Z 
2022-12-01T10:45:06.6764667Z OK (skipped=1)
2022-12-01T10:45:06.6764685Z 
2022-12-01T10:45:06.6764813Z Generating XML reports...
2022-12-01T10:45:06.6765265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103739.xml
2022-12-01T10:45:06.6765637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6765814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6766196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6766391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6766410Z 
2022-12-01T10:45:06.6766519Z Running tests...
2022-12-01T10:45:06.6766765Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6767081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6767382Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6767610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43367
2022-12-01T10:45:06.6767829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43368
2022-12-01T10:45:06.6768204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6768381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6768762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6768954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6769305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6769483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6769866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6770058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6770304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6770551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6770950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6771353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6771586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6771852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6771956Z ok (7.098s)
2022-12-01T10:45:06.6771976Z 
2022-12-01T10:45:06.6772243Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6772357Z Ran 1 test in 7.098s
2022-12-01T10:45:06.6772376Z 
2022-12-01T10:45:06.6772471Z OK
2022-12-01T10:45:06.6772489Z 
2022-12-01T10:45:06.6772616Z Generating XML reports...
2022-12-01T10:45:06.6773069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103741.xml
2022-12-01T10:45:06.6773440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6773597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6773978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6774177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6774197Z 
2022-12-01T10:45:06.6774308Z Running tests...
2022-12-01T10:45:06.6774571Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6774885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6775154Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6775908Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.801s)
2022-12-01T10:45:06.6775928Z 
2022-12-01T10:45:06.6776194Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6776310Z Ran 1 test in 1.801s
2022-12-01T10:45:06.6776330Z 
2022-12-01T10:45:06.6776461Z OK (skipped=1)
2022-12-01T10:45:06.6776482Z 
2022-12-01T10:45:06.6776611Z Generating XML reports...
2022-12-01T10:45:06.6777063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103751.xml
2022-12-01T10:45:06.6777437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6777613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6777995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6778188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6778208Z 
2022-12-01T10:45:06.6778321Z Running tests...
2022-12-01T10:45:06.6778567Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6778893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6779144Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6779416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43812
2022-12-01T10:45:06.6779640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43813
2022-12-01T10:45:06.6780012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6780189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6780571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6780766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6781112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6781346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6781729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6781920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6782169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6782414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6782819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6783219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6783431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6783681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6783912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6784149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6784542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6784937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6785040Z ok (4.128s)
2022-12-01T10:45:06.6785060Z 
2022-12-01T10:45:06.6785327Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6785442Z Ran 1 test in 4.128s
2022-12-01T10:45:06.6785462Z 
2022-12-01T10:45:06.6785543Z OK
2022-12-01T10:45:06.6785582Z 
2022-12-01T10:45:06.6785687Z Generating XML reports...
2022-12-01T10:45:06.6786184Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103755.xml
2022-12-01T10:45:06.6786571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6786750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6787134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6787327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6787346Z 
2022-12-01T10:45:06.6787457Z Running tests...
2022-12-01T10:45:06.6787722Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6788022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6788279Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6788500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44129
2022-12-01T10:45:06.6788718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44130
2022-12-01T10:45:06.6789089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6789267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6789876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6790070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6790422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6790688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6791074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6791264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6791513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6791756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6792157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6792555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6792790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6793004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6793110Z ok (6.782s)
2022-12-01T10:45:06.6793130Z 
2022-12-01T10:45:06.6793402Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6793515Z Ran 1 test in 6.782s
2022-12-01T10:45:06.6793534Z 
2022-12-01T10:45:06.6793628Z OK
2022-12-01T10:45:06.6793647Z 
2022-12-01T10:45:06.6793772Z Generating XML reports...
2022-12-01T10:45:06.6794224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103801.xml
2022-12-01T10:45:06.6794594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6794753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6795136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6795333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6795353Z 
2022-12-01T10:45:06.6795521Z Running tests...
2022-12-01T10:45:06.6795793Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6796107Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6796354Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6796575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44434
2022-12-01T10:45:06.6796796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44435
2022-12-01T10:45:06.6797156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6797333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6797713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6797913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6798279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6798457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6798829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6799019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6799246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6799493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6799896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6800353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6800585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6800816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6800921Z ok (4.141s)
2022-12-01T10:45:06.6800940Z 
2022-12-01T10:45:06.6801206Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6801321Z Ran 1 test in 4.141s
2022-12-01T10:45:06.6801341Z 
2022-12-01T10:45:06.6801416Z OK
2022-12-01T10:45:06.6801435Z 
2022-12-01T10:45:06.6801562Z Generating XML reports...
2022-12-01T10:45:06.6802018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103810.xml
2022-12-01T10:45:06.6802390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6802577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6802960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6803155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6803174Z 
2022-12-01T10:45:06.6803286Z Running tests...
2022-12-01T10:45:06.6803549Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6803847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6804118Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6804341Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44723
2022-12-01T10:45:06.6804561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44724
2022-12-01T10:45:06.6804981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6805165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6805541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6805735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6806080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6806256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6806629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6806822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6807077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6807325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6807727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6808121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6808350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6808573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6808802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6809042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6809520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6809913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6810017Z ok (4.120s)
2022-12-01T10:45:06.6810037Z 
2022-12-01T10:45:06.6810302Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6810415Z Ran 1 test in 4.120s
2022-12-01T10:45:06.6810434Z 
2022-12-01T10:45:06.6810528Z OK
2022-12-01T10:45:06.6810547Z 
2022-12-01T10:45:06.6810653Z Generating XML reports...
2022-12-01T10:45:06.6811104Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103816.xml
2022-12-01T10:45:06.6811478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6811664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6812043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6812237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6812257Z 
2022-12-01T10:45:06.6812366Z Running tests...
2022-12-01T10:45:06.6812629Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6812926Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6813188Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6813411Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45038
2022-12-01T10:45:06.6813629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45039
2022-12-01T10:45:06.6814053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6814238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6814621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6814814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6815182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6815340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6815723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6815914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6816163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6816418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6816819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6817221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6817457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6817701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6817907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6818144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6818549Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6818996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6819103Z ok (4.212s)
2022-12-01T10:45:06.6819123Z 
2022-12-01T10:45:06.6819390Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6819506Z Ran 1 test in 4.213s
2022-12-01T10:45:06.6819526Z 
2022-12-01T10:45:06.6819622Z OK
2022-12-01T10:45:06.6819641Z 
2022-12-01T10:45:06.6819747Z Generating XML reports...
2022-12-01T10:45:06.6820200Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103823.xml
2022-12-01T10:45:06.6820573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6820751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6821140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6821333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6821353Z 
2022-12-01T10:45:06.6821464Z Running tests...
2022-12-01T10:45:06.6821731Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6822048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6822296Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6822516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45294
2022-12-01T10:45:06.6822734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45295
2022-12-01T10:45:06.6823107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6823333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6823724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6823918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6824284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6824438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6824820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6825010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6825258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6825508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6825914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6826317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6826550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6826780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6827018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj03ze2qw
2022-12-01T10:45:06.6827290Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj03ze2qw/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6827548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqz7qoseg
2022-12-01T10:45:06.6827876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqz7qoseg/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6827980Z ok (6.161s)
2022-12-01T10:45:06.6828000Z 
2022-12-01T10:45:06.6828268Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6828383Z Ran 1 test in 6.162s
2022-12-01T10:45:06.6828402Z 
2022-12-01T10:45:06.6828500Z OK
2022-12-01T10:45:06.6828519Z 
2022-12-01T10:45:06.6828644Z Generating XML reports...
2022-12-01T10:45:06.6829076Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103829.xml
2022-12-01T10:45:06.6829705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6829892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6830285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6830487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6830507Z 
2022-12-01T10:45:06.6830618Z Running tests...
2022-12-01T10:45:06.6830886Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6831203Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6831424Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-12-01T10:45:06.6831462Z 
2022-12-01T10:45:06.6831706Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6831818Z Ran 1 test in 0.003s
2022-12-01T10:45:06.6831837Z 
2022-12-01T10:45:06.6831946Z OK (skipped=1)
2022-12-01T10:45:06.6831965Z 
2022-12-01T10:45:06.6832090Z Generating XML reports...
2022-12-01T10:45:06.6832543Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103838.xml
2022-12-01T10:45:06.6832995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6833182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6833566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6833741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6833761Z 
2022-12-01T10:45:06.6833869Z Running tests...
2022-12-01T10:45:06.6834133Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6834448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6834687Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.003s)
2022-12-01T10:45:06.6834706Z 
2022-12-01T10:45:06.6834970Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6835089Z Ran 1 test in 0.003s
2022-12-01T10:45:06.6835108Z 
2022-12-01T10:45:06.6835222Z OK (skipped=1)
2022-12-01T10:45:06.6835241Z 
2022-12-01T10:45:06.6835367Z Generating XML reports...
2022-12-01T10:45:06.6835799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103840.xml
2022-12-01T10:45:06.6836168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6836348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6836732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6836923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6836942Z 
2022-12-01T10:45:06.6837052Z Running tests...
2022-12-01T10:45:06.6837317Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6837704Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6837955Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-12-01T10:45:06.6837998Z 
2022-12-01T10:45:06.6838238Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6838350Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6838370Z 
2022-12-01T10:45:06.6838476Z OK (skipped=1)
2022-12-01T10:45:06.6838495Z 
2022-12-01T10:45:06.6838619Z Generating XML reports...
2022-12-01T10:45:06.6839068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103842.xml
2022-12-01T10:45:06.6839438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6839615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6840004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6840180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6840199Z 
2022-12-01T10:45:06.6840307Z Running tests...
2022-12-01T10:45:06.6840572Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6840886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6841145Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-12-01T10:45:06.6841165Z 
2022-12-01T10:45:06.6841426Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6841539Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6841558Z 
2022-12-01T10:45:06.6841666Z OK (skipped=1)
2022-12-01T10:45:06.6841685Z 
2022-12-01T10:45:06.6841809Z Generating XML reports...
2022-12-01T10:45:06.6842289Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103844.xml
2022-12-01T10:45:06.6842672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6842849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6843227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6843421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6843441Z 
2022-12-01T10:45:06.6843550Z Running tests...
2022-12-01T10:45:06.6843813Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6844127Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6844392Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6844624Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45950
2022-12-01T10:45:06.6844846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45951
2022-12-01T10:45:06.6845221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6845397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6845778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6845969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6846333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6846506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6846930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6847120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6847368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6847614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6848015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6848414Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6848647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6848891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6849124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6849344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6849747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6850139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6850380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.6850620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.6851005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6851443Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6851688Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:45:06.6851793Z ok (22.861s)
2022-12-01T10:45:06.6851812Z 
2022-12-01T10:45:06.6852064Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6852181Z Ran 1 test in 22.861s
2022-12-01T10:45:06.6852200Z 
2022-12-01T10:45:06.6852293Z OK
2022-12-01T10:45:06.6852312Z 
2022-12-01T10:45:06.6852437Z Generating XML reports...
2022-12-01T10:45:06.6852887Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103846.xml
2022-12-01T10:45:06.6853259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6853438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6853829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6854001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6854039Z 
2022-12-01T10:45:06.6854130Z Running tests...
2022-12-01T10:45:06.6854396Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6854709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6855012Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6855233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46557
2022-12-01T10:45:06.6855451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46558
2022-12-01T10:45:06.6855821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6856067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6856437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6856629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6856999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6857175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6857550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6857738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6857985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6858230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6858624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6859024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6859257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6859501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.6859725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6859964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.6860362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6860804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.6861056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.6861299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.6861679Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6862072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.6862306Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:45:06.6862540Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:45:06.6862646Z ok (22.822s)
2022-12-01T10:45:06.6862670Z 
2022-12-01T10:45:06.6862942Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6863060Z Ran 1 test in 22.822s
2022-12-01T10:45:06.6863080Z 
2022-12-01T10:45:06.6863175Z OK
2022-12-01T10:45:06.6863194Z 
2022-12-01T10:45:06.6863300Z Generating XML reports...
2022-12-01T10:45:06.6863757Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103911.xml
2022-12-01T10:45:06.6864135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6864317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6864697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6864889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6864908Z 
2022-12-01T10:45:06.6865072Z Running tests...
2022-12-01T10:45:06.6865341Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6865654Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6866044Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.6866064Z 
2022-12-01T10:45:06.6866328Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6866440Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6866460Z 
2022-12-01T10:45:06.6866571Z OK (skipped=1)
2022-12-01T10:45:06.6866589Z 
2022-12-01T10:45:06.6866713Z Generating XML reports...
2022-12-01T10:45:06.6867161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103936.xml
2022-12-01T10:45:06.6867537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6867722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6868106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6868281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6868300Z 
2022-12-01T10:45:06.6868408Z Running tests...
2022-12-01T10:45:06.6868674Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6868988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6869590Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.6869612Z 
2022-12-01T10:45:06.6869888Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6870004Z Ran 1 test in 0.003s
2022-12-01T10:45:06.6870029Z 
2022-12-01T10:45:06.6870138Z OK (skipped=1)
2022-12-01T10:45:06.6870157Z 
2022-12-01T10:45:06.6870355Z Generating XML reports...
2022-12-01T10:45:06.6870805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103938.xml
2022-12-01T10:45:06.6871179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6871356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6871738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6871935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6871954Z 
2022-12-01T10:45:06.6872066Z Running tests...
2022-12-01T10:45:06.6872334Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6872652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6873064Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.6873103Z 
2022-12-01T10:45:06.6873343Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6873457Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6873477Z 
2022-12-01T10:45:06.6873587Z OK (skipped=1)
2022-12-01T10:45:06.6873606Z 
2022-12-01T10:45:06.6873732Z Generating XML reports...
2022-12-01T10:45:06.6874180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103940.xml
2022-12-01T10:45:06.6874554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6874731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6875114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6875357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6875395Z 
2022-12-01T10:45:06.6875486Z Running tests...
2022-12-01T10:45:06.6875752Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6876070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6876484Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.6876504Z 
2022-12-01T10:45:06.6876767Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6876883Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6876902Z 
2022-12-01T10:45:06.6877011Z OK (skipped=1)
2022-12-01T10:45:06.6877030Z 
2022-12-01T10:45:06.6877154Z Generating XML reports...
2022-12-01T10:45:06.6877590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103942.xml
2022-12-01T10:45:06.6877968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6878145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6878531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6878723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6878742Z 
2022-12-01T10:45:06.6878851Z Running tests...
2022-12-01T10:45:06.6879115Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6879477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6879875Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.6879917Z 
2022-12-01T10:45:06.6880209Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6880333Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6880352Z 
2022-12-01T10:45:06.6880459Z OK (skipped=1)
2022-12-01T10:45:06.6880478Z 
2022-12-01T10:45:06.6880605Z Generating XML reports...
2022-12-01T10:45:06.6881061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103944.xml
2022-12-01T10:45:06.6881430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6881608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6881988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6882161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6882204Z 
2022-12-01T10:45:06.6882294Z Running tests...
2022-12-01T10:45:06.6882563Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6882877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6883153Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6883376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47914
2022-12-01T10:45:06.6883594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47915
2022-12-01T10:45:06.6883969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6884146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6884510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6884757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6885129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6885302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6885678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6885871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6886122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6886369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6886752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6887158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6887392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6887623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6887727Z ok (5.764s)
2022-12-01T10:45:06.6887746Z 
2022-12-01T10:45:06.6888013Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6888128Z Ran 1 test in 5.764s
2022-12-01T10:45:06.6888148Z 
2022-12-01T10:45:06.6888249Z OK
2022-12-01T10:45:06.6888269Z 
2022-12-01T10:45:06.6888397Z Generating XML reports...
2022-12-01T10:45:06.6888833Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103946.xml
2022-12-01T10:45:06.6889212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6889437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6889829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6890024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6890043Z 
2022-12-01T10:45:06.6890154Z Running tests...
2022-12-01T10:45:06.6890418Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6890739Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6891016Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6891220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48247
2022-12-01T10:45:06.6891439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48248
2022-12-01T10:45:06.6891820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6892001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6892389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6892581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6892948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6893127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6893484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6893675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6893977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6894226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6894629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6895028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6895259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6895493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6895599Z ok (5.690s)
2022-12-01T10:45:06.6895618Z 
2022-12-01T10:45:06.6895867Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6895982Z Ran 1 test in 5.690s
2022-12-01T10:45:06.6896006Z 
2022-12-01T10:45:06.6896101Z OK
2022-12-01T10:45:06.6896119Z 
2022-12-01T10:45:06.6896247Z Generating XML reports...
2022-12-01T10:45:06.6896695Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103954.xml
2022-12-01T10:45:06.6897067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6897243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6897628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6897805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6897844Z 
2022-12-01T10:45:06.6897936Z Running tests...
2022-12-01T10:45:06.6898204Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6898519Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6898842Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6899069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48678
2022-12-01T10:45:06.6899288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48679
2022-12-01T10:45:06.6899662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6899838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6900198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6900391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6900755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6900933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6901311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6901502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6901748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6901994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6902374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6902776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6903009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6903326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6903431Z ok (6.634s)
2022-12-01T10:45:06.6903450Z 
2022-12-01T10:45:06.6903716Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6903831Z Ran 1 test in 6.634s
2022-12-01T10:45:06.6903851Z 
2022-12-01T10:45:06.6903946Z OK
2022-12-01T10:45:06.6903964Z 
2022-12-01T10:45:06.6904091Z Generating XML reports...
2022-12-01T10:45:06.6904525Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104002.xml
2022-12-01T10:45:06.6904897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6905076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6905461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6905663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6905682Z 
2022-12-01T10:45:06.6905793Z Running tests...
2022-12-01T10:45:06.6906056Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6906372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6906643Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6906846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49056
2022-12-01T10:45:06.6907065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49057
2022-12-01T10:45:06.6907435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6907611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6908038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6908237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6908605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6908781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6909134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6909325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6909799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6910049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6910462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6910869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6911115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6911536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6911722Z ok (5.520s)
2022-12-01T10:45:06.6911751Z 
2022-12-01T10:45:06.6912157Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6912275Z Ran 1 test in 5.520s
2022-12-01T10:45:06.6912295Z 
2022-12-01T10:45:06.6912389Z OK
2022-12-01T10:45:06.6912409Z 
2022-12-01T10:45:06.6912534Z Generating XML reports...
2022-12-01T10:45:06.6912992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104011.xml
2022-12-01T10:45:06.6913485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6913663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6914047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6914222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6914261Z 
2022-12-01T10:45:06.6914352Z Running tests...
2022-12-01T10:45:06.6914616Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6914936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6915227Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6915988Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.732s)
2022-12-01T10:45:06.6916012Z 
2022-12-01T10:45:06.6916273Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6916386Z Ran 1 test in 1.733s
2022-12-01T10:45:06.6916405Z 
2022-12-01T10:45:06.6916514Z OK (skipped=1)
2022-12-01T10:45:06.6916533Z 
2022-12-01T10:45:06.6916658Z Generating XML reports...
2022-12-01T10:45:06.6917109Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104019.xml
2022-12-01T10:45:06.6917461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6917641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6918023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6918279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6918300Z 
2022-12-01T10:45:06.6918415Z Running tests...
2022-12-01T10:45:06.6918680Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6918995Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6919251Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:45:06.6919271Z 
2022-12-01T10:45:06.6919535Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6919631Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6919650Z 
2022-12-01T10:45:06.6919760Z OK (skipped=1)
2022-12-01T10:45:06.6919781Z 
2022-12-01T10:45:06.6919970Z Generating XML reports...
2022-12-01T10:45:06.6920730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104023.xml
2022-12-01T10:45:06.6921126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6921305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6921686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6921879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6921900Z 
2022-12-01T10:45:06.6921990Z Running tests...
2022-12-01T10:45:06.6922256Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6922574Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6922848Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:45:06.6922946Z 
2022-12-01T10:45:06.6923222Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6923336Z Ran 1 test in 0.003s
2022-12-01T10:45:06.6923355Z 
2022-12-01T10:45:06.6923465Z OK (skipped=1)
2022-12-01T10:45:06.6923484Z 
2022-12-01T10:45:06.6923610Z Generating XML reports...
2022-12-01T10:45:06.6924062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104025.xml
2022-12-01T10:45:06.6924417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6924592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6924972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6925166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6925190Z 
2022-12-01T10:45:06.6925300Z Running tests...
2022-12-01T10:45:06.6925566Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6925884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6926197Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:45:06.6926216Z 
2022-12-01T10:45:06.6926477Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6926571Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6926590Z 
2022-12-01T10:45:06.6926697Z OK (skipped=1)
2022-12-01T10:45:06.6926716Z 
2022-12-01T10:45:06.6926841Z Generating XML reports...
2022-12-01T10:45:06.6927290Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104027.xml
2022-12-01T10:45:06.6927664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6927889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6928281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6928475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6928494Z 
2022-12-01T10:45:06.6928604Z Running tests...
2022-12-01T10:45:06.6928850Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6929162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6929471Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6929691Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49789
2022-12-01T10:45:06.6929910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49790
2022-12-01T10:45:06.6930288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6930465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6930846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6931018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6931387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6931563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6931935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6932126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6932430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6932677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6933077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6933477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6933688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6933919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6934021Z ok (4.118s)
2022-12-01T10:45:06.6934041Z 
2022-12-01T10:45:06.6934308Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6934422Z Ran 1 test in 4.119s
2022-12-01T10:45:06.6934445Z 
2022-12-01T10:45:06.6934543Z OK
2022-12-01T10:45:06.6934562Z 
2022-12-01T10:45:06.6934688Z Generating XML reports...
2022-12-01T10:45:06.6935140Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104029.xml
2022-12-01T10:45:06.6935491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6935668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6936048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6936240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6936259Z 
2022-12-01T10:45:06.6936369Z Running tests...
2022-12-01T10:45:06.6936632Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6936948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6937294Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6937520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50104
2022-12-01T10:45:06.6937720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50105
2022-12-01T10:45:06.6938097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6938273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6938654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6938846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6939210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6939395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6939768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6939939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6940185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6940431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6940835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6941234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6941467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6941749Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6941854Z ok (3.981s)
2022-12-01T10:45:06.6941873Z 
2022-12-01T10:45:06.6942139Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6942234Z Ran 1 test in 3.982s
2022-12-01T10:45:06.6942253Z 
2022-12-01T10:45:06.6942347Z OK
2022-12-01T10:45:06.6942366Z 
2022-12-01T10:45:06.6942492Z Generating XML reports...
2022-12-01T10:45:06.6942942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104035.xml
2022-12-01T10:45:06.6943317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6943496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6943877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6944077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6944097Z 
2022-12-01T10:45:06.6944208Z Running tests...
2022-12-01T10:45:06.6944457Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6944773Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6945051Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:45:06.6945070Z 
2022-12-01T10:45:06.6945329Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6945443Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6945462Z 
2022-12-01T10:45:06.6945573Z OK (skipped=1)
2022-12-01T10:45:06.6945591Z 
2022-12-01T10:45:06.6945718Z Generating XML reports...
2022-12-01T10:45:06.6946169Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104042.xml
2022-12-01T10:45:06.6946574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6946756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6947138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6947332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6947353Z 
2022-12-01T10:45:06.6947471Z Running tests...
2022-12-01T10:45:06.6947737Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6948053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6948361Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:45:06.6948384Z 
2022-12-01T10:45:06.6948654Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6948747Z Ran 1 test in 0.002s
2022-12-01T10:45:06.6948789Z 
2022-12-01T10:45:06.6948879Z OK (skipped=1)
2022-12-01T10:45:06.6948898Z 
2022-12-01T10:45:06.6949021Z Generating XML reports...
2022-12-01T10:45:06.6949673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104044.xml
2022-12-01T10:45:06.6950067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6950243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6950625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6950819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6950917Z 
2022-12-01T10:45:06.6951035Z Running tests...
2022-12-01T10:45:06.6951289Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6951606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6951887Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6952635Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.708s)
2022-12-01T10:45:06.6952656Z 
2022-12-01T10:45:06.6952921Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6953034Z Ran 1 test in 1.708s
2022-12-01T10:45:06.6953054Z 
2022-12-01T10:45:06.6953163Z OK (skipped=1)
2022-12-01T10:45:06.6953186Z 
2022-12-01T10:45:06.6953311Z Generating XML reports...
2022-12-01T10:45:06.6953764Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104046.xml
2022-12-01T10:45:06.6954136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6954294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6954674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6954866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6954885Z 
2022-12-01T10:45:06.6954996Z Running tests...
2022-12-01T10:45:06.6955259Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6955575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6955921Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6956155Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50769
2022-12-01T10:45:06.6956355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50770
2022-12-01T10:45:06.6956730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6956906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6957289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6957482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6957848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6958028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6958412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6958604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6958834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6959077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6959483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6959883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6960114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6960392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6960654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsmtggnkx
2022-12-01T10:45:06.6960930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsmtggnkx/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6961165Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nk5mgfg
2022-12-01T10:45:06.6961433Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nk5mgfg/_remote_module_non_scriptable.py
2022-12-01T10:45:06.6961536Z ok (7.454s)
2022-12-01T10:45:06.6961555Z 
2022-12-01T10:45:06.6961830Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6961941Z Ran 1 test in 7.454s
2022-12-01T10:45:06.6961961Z 
2022-12-01T10:45:06.6962059Z OK
2022-12-01T10:45:06.6962080Z 
2022-12-01T10:45:06.6962200Z Generating XML reports...
2022-12-01T10:45:06.6962931Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104050.xml
2022-12-01T10:45:06.6963334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6963493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6963877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6964069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6964089Z 
2022-12-01T10:45:06.6964199Z Running tests...
2022-12-01T10:45:06.6964459Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6964775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6965049Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6965276Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51151
2022-12-01T10:45:06.6965542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51152
2022-12-01T10:45:06.6965927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6966102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6966482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6966672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6967036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6967210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6967589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6967785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6968015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6968262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6968668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6969070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6969299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6969528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6969632Z ok (6.778s)
2022-12-01T10:45:06.6969699Z 
2022-12-01T10:45:06.6969978Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6970074Z Ran 1 test in 6.778s
2022-12-01T10:45:06.6970113Z 
2022-12-01T10:45:06.6970190Z OK
2022-12-01T10:45:06.6970208Z 
2022-12-01T10:45:06.6970334Z Generating XML reports...
2022-12-01T10:45:06.6970785Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104100.xml
2022-12-01T10:45:06.6971159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6971338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6971719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6971911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6971930Z 
2022-12-01T10:45:06.6972045Z Running tests...
2022-12-01T10:45:06.6972294Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6972614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6972903Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6973122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51526
2022-12-01T10:45:06.6973341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51527
2022-12-01T10:45:06.6973709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6973888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6974273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6974469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6974861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6975048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6975425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6975613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6975862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6976106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6976507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6976907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6977124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6977356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6977464Z ok (6.871s)
2022-12-01T10:45:06.6977484Z 
2022-12-01T10:45:06.6977755Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6977871Z Ran 1 test in 6.871s
2022-12-01T10:45:06.6977890Z 
2022-12-01T10:45:06.6977984Z OK
2022-12-01T10:45:06.6978003Z 
2022-12-01T10:45:06.6978128Z Generating XML reports...
2022-12-01T10:45:06.6978582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104109.xml
2022-12-01T10:45:06.6978958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6979171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6979615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6979811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6979830Z 
2022-12-01T10:45:06.6979941Z Running tests...
2022-12-01T10:45:06.6980210Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6980528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6980809Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6981561Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.763s)
2022-12-01T10:45:06.6981586Z 
2022-12-01T10:45:06.6981852Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6981947Z Ran 1 test in 1.763s
2022-12-01T10:45:06.6981988Z 
2022-12-01T10:45:06.6982077Z OK (skipped=1)
2022-12-01T10:45:06.6982096Z 
2022-12-01T10:45:06.6982224Z Generating XML reports...
2022-12-01T10:45:06.6982678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104118.xml
2022-12-01T10:45:06.6983053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6983228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6983613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6983808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6983831Z 
2022-12-01T10:45:06.6983991Z Running tests...
2022-12-01T10:45:06.6984300Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6984618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6984914Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6985658Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.790s)
2022-12-01T10:45:06.6985679Z 
2022-12-01T10:45:06.6985944Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6986059Z Ran 1 test in 1.791s
2022-12-01T10:45:06.6986083Z 
2022-12-01T10:45:06.6986194Z OK (skipped=1)
2022-12-01T10:45:06.6986214Z 
2022-12-01T10:45:06.6986345Z Generating XML reports...
2022-12-01T10:45:06.6986795Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104122.xml
2022-12-01T10:45:06.6987168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6987326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6987708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6987903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6987922Z 
2022-12-01T10:45:06.6988034Z Running tests...
2022-12-01T10:45:06.6988300Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6988616Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6989007Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6989231Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52249
2022-12-01T10:45:06.6989666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52250
2022-12-01T10:45:06.6990064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6990244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6990626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6990822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6991191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6991376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6991755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6991947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6992174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.6992422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.6992827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6993227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.6993460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.6993783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.6993949Z skip: Need at least 4 CUDA devices (3.888s)
2022-12-01T10:45:06.6993970Z 
2022-12-01T10:45:06.6994242Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6994356Z Ran 1 test in 3.888s
2022-12-01T10:45:06.6994375Z 
2022-12-01T10:45:06.6994464Z OK (skipped=1)
2022-12-01T10:45:06.6994483Z 
2022-12-01T10:45:06.6994608Z Generating XML reports...
2022-12-01T10:45:06.6995063Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104126.xml
2022-12-01T10:45:06.6995435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6995615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6995995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6996198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6996217Z 
2022-12-01T10:45:06.6996327Z Running tests...
2022-12-01T10:45:06.6996578Z ----------------------------------------------------------------------
2022-12-01T10:45:06.6996895Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.6997220Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.6997442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52359
2022-12-01T10:45:06.6997660Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52360
2022-12-01T10:45:06.6998031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6998281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6998665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6998858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.6999203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.6999379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.6999761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.6999951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7000199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7000448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7000856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7001252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7001482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7001693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7001846Z skip: Need at least 4 CUDA devices (3.708s)
2022-12-01T10:45:06.7001866Z 
2022-12-01T10:45:06.7002132Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7002247Z Ran 1 test in 3.708s
2022-12-01T10:45:06.7002266Z 
2022-12-01T10:45:06.7002374Z OK (skipped=1)
2022-12-01T10:45:06.7002393Z 
2022-12-01T10:45:06.7002524Z Generating XML reports...
2022-12-01T10:45:06.7003020Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104132.xml
2022-12-01T10:45:06.7003405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7003582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7003945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7004137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7004156Z 
2022-12-01T10:45:06.7004265Z Running tests...
2022-12-01T10:45:06.7004533Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7004847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7005136Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7005892Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.590s)
2022-12-01T10:45:06.7005913Z 
2022-12-01T10:45:06.7006177Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7006290Z Ran 1 test in 1.591s
2022-12-01T10:45:06.7006309Z 
2022-12-01T10:45:06.7006398Z OK (skipped=1)
2022-12-01T10:45:06.7006437Z 
2022-12-01T10:45:06.7006543Z Generating XML reports...
2022-12-01T10:45:06.7006994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104138.xml
2022-12-01T10:45:06.7007368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7007602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7007986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7008178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7008197Z 
2022-12-01T10:45:06.7008305Z Running tests...
2022-12-01T10:45:06.7008568Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7008863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7009136Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7009156Z 
2022-12-01T10:45:06.7009419Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7009534Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7009558Z 
2022-12-01T10:45:06.7009668Z OK (skipped=1)
2022-12-01T10:45:06.7009687Z 
2022-12-01T10:45:06.7009815Z Generating XML reports...
2022-12-01T10:45:06.7010265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104141.xml
2022-12-01T10:45:06.7010638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7010813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7011172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7011368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7011387Z 
2022-12-01T10:45:06.7011497Z Running tests...
2022-12-01T10:45:06.7011763Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7012079Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7012396Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7012417Z 
2022-12-01T10:45:06.7012684Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7012798Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7012817Z 
2022-12-01T10:45:06.7012906Z OK (skipped=1)
2022-12-01T10:45:06.7012945Z 
2022-12-01T10:45:06.7013051Z Generating XML reports...
2022-12-01T10:45:06.7013500Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104143.xml
2022-12-01T10:45:06.7013870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7014049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7014423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7014623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7014642Z 
2022-12-01T10:45:06.7014752Z Running tests...
2022-12-01T10:45:06.7015014Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7015308Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7015584Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7015604Z 
2022-12-01T10:45:06.7015865Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7015978Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7015997Z 
2022-12-01T10:45:06.7016108Z OK (skipped=1)
2022-12-01T10:45:06.7016127Z 
2022-12-01T10:45:06.7016253Z Generating XML reports...
2022-12-01T10:45:06.7016699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104145.xml
2022-12-01T10:45:06.7017132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7017308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7017666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7017859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7017879Z 
2022-12-01T10:45:06.7017990Z Running tests...
2022-12-01T10:45:06.7018253Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7018566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7018836Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7018860Z 
2022-12-01T10:45:06.7019123Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7019237Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7019257Z 
2022-12-01T10:45:06.7019365Z OK (skipped=1)
2022-12-01T10:45:06.7019384Z 
2022-12-01T10:45:06.7019489Z Generating XML reports...
2022-12-01T10:45:06.7019935Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104147.xml
2022-12-01T10:45:06.7020305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7020481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7020856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7021049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7021068Z 
2022-12-01T10:45:06.7021183Z Running tests...
2022-12-01T10:45:06.7021492Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7021795Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7022059Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-12-01T10:45:06.7022078Z 
2022-12-01T10:45:06.7022339Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7022453Z Ran 1 test in 0.003s
2022-12-01T10:45:06.7022472Z 
2022-12-01T10:45:06.7022579Z OK (skipped=1)
2022-12-01T10:45:06.7022598Z 
2022-12-01T10:45:06.7022722Z Generating XML reports...
2022-12-01T10:45:06.7023175Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104149.xml
2022-12-01T10:45:06.7023548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7023727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7024095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7024286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7024305Z 
2022-12-01T10:45:06.7024412Z Running tests...
2022-12-01T10:45:06.7024675Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7024988Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7025250Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7025270Z 
2022-12-01T10:45:06.7025530Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7025644Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7025663Z 
2022-12-01T10:45:06.7025823Z OK (skipped=1)
2022-12-01T10:45:06.7025844Z 
2022-12-01T10:45:06.7025949Z Generating XML reports...
2022-12-01T10:45:06.7026400Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104151.xml
2022-12-01T10:45:06.7026773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7026948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7027328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7027519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7027538Z 
2022-12-01T10:45:06.7027648Z Running tests...
2022-12-01T10:45:06.7027914Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7028209Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7028486Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7028506Z 
2022-12-01T10:45:06.7028767Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7028880Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7028899Z 
2022-12-01T10:45:06.7029008Z OK (skipped=1)
2022-12-01T10:45:06.7029027Z 
2022-12-01T10:45:06.7029151Z Generating XML reports...
2022-12-01T10:45:06.7029919Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104153.xml
2022-12-01T10:45:06.7030311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7030488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7030851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7031121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7031142Z 
2022-12-01T10:45:06.7031258Z Running tests...
2022-12-01T10:45:06.7031530Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7031846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7032111Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7032131Z 
2022-12-01T10:45:06.7032390Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7032507Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7032526Z 
2022-12-01T10:45:06.7032636Z OK (skipped=1)
2022-12-01T10:45:06.7032655Z 
2022-12-01T10:45:06.7032761Z Generating XML reports...
2022-12-01T10:45:06.7033214Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104155.xml
2022-12-01T10:45:06.7033591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7033767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7034144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7034335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7034354Z 
2022-12-01T10:45:06.7034464Z Running tests...
2022-12-01T10:45:06.7034731Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7035028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7035287Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7035306Z 
2022-12-01T10:45:06.7035568Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7035746Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7035768Z 
2022-12-01T10:45:06.7035877Z OK (skipped=1)
2022-12-01T10:45:06.7035896Z 
2022-12-01T10:45:06.7036019Z Generating XML reports...
2022-12-01T10:45:06.7036467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104157.xml
2022-12-01T10:45:06.7036838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7037016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7037374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7037566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7037585Z 
2022-12-01T10:45:06.7037696Z Running tests...
2022-12-01T10:45:06.7037963Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7038284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7038540Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7038560Z 
2022-12-01T10:45:06.7038821Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7038933Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7038952Z 
2022-12-01T10:45:06.7039065Z OK (skipped=1)
2022-12-01T10:45:06.7039084Z 
2022-12-01T10:45:06.7039188Z Generating XML reports...
2022-12-01T10:45:06.7039631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104159.xml
2022-12-01T10:45:06.7040001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7040178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7040605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7040802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7040821Z 
2022-12-01T10:45:06.7040930Z Running tests...
2022-12-01T10:45:06.7041197Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7041493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7041752Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7041974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52955
2022-12-01T10:45:06.7042194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52956
2022-12-01T10:45:06.7042567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7042754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7043138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7043329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7043693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7043850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7044234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7044425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7044678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7044975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7045382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7045782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7046013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7046242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7046562Z STAGE:2022-12-01 10:42:06 52955:52955 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7047330Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1640: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:45:06.7047454Z   warnings.warn(
2022-12-01T10:45:06.7047795Z STAGE:2022-12-01 10:42:10 52956:52956 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7048548Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1640: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:45:06.7048661Z   warnings.warn(
2022-12-01T10:45:06.7049211Z STAGE:2022-12-01 10:42:11 52956:52956 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:42:11 52955:52955 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7049233Z 
2022-12-01T10:45:06.7049567Z STAGE:2022-12-01 10:42:11 52956:52956 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7049991Z STAGE:2022-12-01 10:42:11 52956:52956 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7050333Z STAGE:2022-12-01 10:42:11 52955:52955 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7050643Z STAGE:2022-12-01 10:42:11 52955:52955 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7050746Z ok (10.220s)
2022-12-01T10:45:06.7050766Z 
2022-12-01T10:45:06.7051028Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7051143Z Ran 1 test in 10.220s
2022-12-01T10:45:06.7051163Z 
2022-12-01T10:45:06.7051257Z OK
2022-12-01T10:45:06.7051276Z 
2022-12-01T10:45:06.7051402Z Generating XML reports...
2022-12-01T10:45:06.7051854Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104201.xml
2022-12-01T10:45:06.7052226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7052416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7052778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7052974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7052994Z 
2022-12-01T10:45:06.7053105Z Running tests...
2022-12-01T10:45:06.7053371Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7053687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7053957Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7053977Z 
2022-12-01T10:45:06.7054240Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7054356Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7054431Z 
2022-12-01T10:45:06.7054530Z OK (skipped=1)
2022-12-01T10:45:06.7054569Z 
2022-12-01T10:45:06.7054678Z Generating XML reports...
2022-12-01T10:45:06.7055137Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104213.xml
2022-12-01T10:45:06.7055508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7055687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7056067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7056261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7056280Z 
2022-12-01T10:45:06.7056390Z Running tests...
2022-12-01T10:45:06.7056658Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7056954Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7057234Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7057457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53248
2022-12-01T10:45:06.7057674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53249
2022-12-01T10:45:06.7058045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7058222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7058603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7058794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7059163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7059325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7059753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7059950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7060200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7060444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7060848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7061246Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7061476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7061688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7062028Z STAGE:2022-12-01 10:42:20 53248:53248 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7062353Z STAGE:2022-12-01 10:42:21 53249:53249 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7062689Z STAGE:2022-12-01 10:42:22 53248:53248 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7063020Z STAGE:2022-12-01 10:42:22 53249:53249 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7063351Z STAGE:2022-12-01 10:42:22 53249:53249 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7063682Z STAGE:2022-12-01 10:42:22 53249:53249 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7064008Z STAGE:2022-12-01 10:42:22 53248:53248 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7064396Z STAGE:2022-12-01 10:42:22 53248:53248 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7064481Z ok (7.607s)
2022-12-01T10:45:06.7064519Z 
2022-12-01T10:45:06.7064766Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7064880Z Ran 1 test in 7.607s
2022-12-01T10:45:06.7064900Z 
2022-12-01T10:45:06.7064995Z OK
2022-12-01T10:45:06.7065015Z 
2022-12-01T10:45:06.7065139Z Generating XML reports...
2022-12-01T10:45:06.7065596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104215.xml
2022-12-01T10:45:06.7065971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7066148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7066532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7066714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7066734Z 
2022-12-01T10:45:06.7066846Z Running tests...
2022-12-01T10:45:06.7067114Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7067431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7067696Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7067917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53478
2022-12-01T10:45:06.7068137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53479
2022-12-01T10:45:06.7068511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7068670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7069112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7069311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7070005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7070187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7070570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7070763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7071011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7071257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7071650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7072050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7072281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7072510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7072613Z ok (6.634s)
2022-12-01T10:45:06.7072634Z 
2022-12-01T10:45:06.7072900Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7073014Z Ran 1 test in 6.634s
2022-12-01T10:45:06.7073034Z 
2022-12-01T10:45:06.7073128Z OK
2022-12-01T10:45:06.7073147Z 
2022-12-01T10:45:06.7073253Z Generating XML reports...
2022-12-01T10:45:06.7073705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104225.xml
2022-12-01T10:45:06.7074175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7074353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7074735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7074927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7074946Z 
2022-12-01T10:45:06.7075057Z Running tests...
2022-12-01T10:45:06.7075320Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7075638Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7075876Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7075896Z 
2022-12-01T10:45:06.7076162Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7076279Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7076301Z 
2022-12-01T10:45:06.7076413Z OK (skipped=1)
2022-12-01T10:45:06.7076432Z 
2022-12-01T10:45:06.7076556Z Generating XML reports...
2022-12-01T10:45:06.7077010Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104234.xml
2022-12-01T10:45:06.7077381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7077559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7077942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7078118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7078137Z 
2022-12-01T10:45:06.7078247Z Running tests...
2022-12-01T10:45:06.7078514Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7078898Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7079168Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7079392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53855
2022-12-01T10:45:06.7079666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53856
2022-12-01T10:45:06.7080047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7080207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7080579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7080756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7081146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7081342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7081726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7081920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7082171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7082419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7082805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7083206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7083493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7083723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7084066Z STAGE:2022-12-01 10:42:41 53856:53856 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7084392Z STAGE:2022-12-01 10:42:41 53855:53855 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7084730Z STAGE:2022-12-01 10:42:41 53856:53856 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7085068Z STAGE:2022-12-01 10:42:41 53855:53855 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7085398Z STAGE:2022-12-01 10:42:41 53855:53855 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7085712Z STAGE:2022-12-01 10:42:41 53855:53855 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7086051Z STAGE:2022-12-01 10:42:41 53856:53856 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7086385Z STAGE:2022-12-01 10:42:41 53856:53856 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7086489Z ok (6.051s)
2022-12-01T10:45:06.7086508Z 
2022-12-01T10:45:06.7086776Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7086890Z Ran 1 test in 6.051s
2022-12-01T10:45:06.7086910Z 
2022-12-01T10:45:06.7087006Z OK
2022-12-01T10:45:06.7087025Z 
2022-12-01T10:45:06.7087149Z Generating XML reports...
2022-12-01T10:45:06.7087586Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104236.xml
2022-12-01T10:45:06.7087957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7088136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7088575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7088775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7088794Z 
2022-12-01T10:45:06.7088904Z Running tests...
2022-12-01T10:45:06.7089170Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7089487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7089753Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7089957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54115
2022-12-01T10:45:06.7090174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54116
2022-12-01T10:45:06.7090548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7090733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7091116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7091311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7091682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7091858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7092216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7092405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7092650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7092893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7093353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7093749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7093985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7094217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7094558Z STAGE:2022-12-01 10:42:49 54116:54116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7094866Z STAGE:2022-12-01 10:42:49 54115:54115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7095208Z STAGE:2022-12-01 10:42:49 54115:54115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7095551Z STAGE:2022-12-01 10:42:49 54116:54116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7095880Z STAGE:2022-12-01 10:42:50 54115:54115 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7096202Z STAGE:2022-12-01 10:42:50 54116:54116 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7096533Z STAGE:2022-12-01 10:42:50 54115:54115 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7096862Z STAGE:2022-12-01 10:42:50 54116:54116 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7096969Z ok (6.060s)
2022-12-01T10:45:06.7096989Z 
2022-12-01T10:45:06.7097255Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7097350Z Ran 1 test in 6.060s
2022-12-01T10:45:06.7097369Z 
2022-12-01T10:45:06.7097463Z OK
2022-12-01T10:45:06.7097483Z 
2022-12-01T10:45:06.7097609Z Generating XML reports...
2022-12-01T10:45:06.7098110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104244.xml
2022-12-01T10:45:06.7098492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7098668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7099049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7099245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7099265Z 
2022-12-01T10:45:06.7099355Z Running tests...
2022-12-01T10:45:06.7099626Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7099938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7100202Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7100226Z 
2022-12-01T10:45:06.7100490Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7100603Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7100622Z 
2022-12-01T10:45:06.7100731Z OK (skipped=1)
2022-12-01T10:45:06.7100750Z 
2022-12-01T10:45:06.7100875Z Generating XML reports...
2022-12-01T10:45:06.7101325Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104252.xml
2022-12-01T10:45:06.7101679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7101856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7102236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7102429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7102495Z 
2022-12-01T10:45:06.7102609Z Running tests...
2022-12-01T10:45:06.7102879Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7103192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7103447Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7103467Z 
2022-12-01T10:45:06.7103732Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7103824Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7103846Z 
2022-12-01T10:45:06.7103955Z OK (skipped=1)
2022-12-01T10:45:06.7103974Z 
2022-12-01T10:45:06.7104098Z Generating XML reports...
2022-12-01T10:45:06.7104548Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104254.xml
2022-12-01T10:45:06.7104917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7105101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7105481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7105674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7105693Z 
2022-12-01T10:45:06.7105804Z Running tests...
2022-12-01T10:45:06.7106055Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7106371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7106632Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7106651Z 
2022-12-01T10:45:06.7106910Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7107022Z Ran 1 test in 0.003s
2022-12-01T10:45:06.7107045Z 
2022-12-01T10:45:06.7107153Z OK (skipped=1)
2022-12-01T10:45:06.7107172Z 
2022-12-01T10:45:06.7107340Z Generating XML reports...
2022-12-01T10:45:06.7107794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104256.xml
2022-12-01T10:45:06.7108150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7108326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7108709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7108901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7108921Z 
2022-12-01T10:45:06.7109030Z Running tests...
2022-12-01T10:45:06.7109294Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7109914Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7110190Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7110210Z 
2022-12-01T10:45:06.7110471Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7110570Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7110589Z 
2022-12-01T10:45:06.7110696Z OK (skipped=1)
2022-12-01T10:45:06.7110716Z 
2022-12-01T10:45:06.7110843Z Generating XML reports...
2022-12-01T10:45:06.7111293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104258.xml
2022-12-01T10:45:06.7111663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7111843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7112224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7112500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7112520Z 
2022-12-01T10:45:06.7112628Z Running tests...
2022-12-01T10:45:06.7112879Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7113191Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7113445Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7113667Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54600
2022-12-01T10:45:06.7113886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54601
2022-12-01T10:45:06.7114260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7114437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7114827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7115003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7115375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7115549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7115929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7116118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7116366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7116611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7117078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7117492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7117706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7117941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7118284Z STAGE:2022-12-01 10:43:05 54600:54600 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7118612Z STAGE:2022-12-01 10:43:06 54601:54601 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7118947Z STAGE:2022-12-01 10:43:07 54600:54600 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7119282Z STAGE:2022-12-01 10:43:07 54601:54601 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7119621Z STAGE:2022-12-01 10:43:07 54600:54600 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7119947Z STAGE:2022-12-01 10:43:07 54601:54601 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7120278Z STAGE:2022-12-01 10:43:07 54601:54601 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7120587Z STAGE:2022-12-01 10:43:07 54600:54600 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7120694Z ok (7.632s)
2022-12-01T10:45:06.7120713Z 
2022-12-01T10:45:06.7120980Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7121095Z Ran 1 test in 7.632s
2022-12-01T10:45:06.7121114Z 
2022-12-01T10:45:06.7121209Z OK
2022-12-01T10:45:06.7121228Z 
2022-12-01T10:45:06.7121355Z Generating XML reports...
2022-12-01T10:45:06.7121808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104300.xml
2022-12-01T10:45:06.7122254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7122415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7122800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7122995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7123015Z 
2022-12-01T10:45:06.7123125Z Running tests...
2022-12-01T10:45:06.7123393Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7123709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7123977Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7124199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54938
2022-12-01T10:45:06.7124406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54939
2022-12-01T10:45:06.7124780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7124955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7125335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7125530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7125893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7126074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7126450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7126692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7126927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7127173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7127579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7127978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7128209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7128440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7128778Z STAGE:2022-12-01 10:43:15 54938:54938 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7129115Z STAGE:2022-12-01 10:43:15 54939:54939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7129453Z STAGE:2022-12-01 10:43:17 54938:54938 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7129769Z STAGE:2022-12-01 10:43:17 54939:54939 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7130099Z STAGE:2022-12-01 10:43:17 54938:54938 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7130423Z STAGE:2022-12-01 10:43:17 54939:54939 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7130757Z STAGE:2022-12-01 10:43:17 54939:54939 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7131083Z STAGE:2022-12-01 10:43:17 54938:54938 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7131188Z ok (7.672s)
2022-12-01T10:45:06.7131207Z 
2022-12-01T10:45:06.7131525Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7131644Z Ran 1 test in 7.672s
2022-12-01T10:45:06.7131664Z 
2022-12-01T10:45:06.7131738Z OK
2022-12-01T10:45:06.7131778Z 
2022-12-01T10:45:06.7131886Z Generating XML reports...
2022-12-01T10:45:06.7132345Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104310.xml
2022-12-01T10:45:06.7132719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7132896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7133280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7133474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7133494Z 
2022-12-01T10:45:06.7133604Z Running tests...
2022-12-01T10:45:06.7133872Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7134173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7134442Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7134463Z 
2022-12-01T10:45:06.7134729Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7134842Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7134861Z 
2022-12-01T10:45:06.7134972Z OK (skipped=1)
2022-12-01T10:45:06.7134991Z 
2022-12-01T10:45:06.7135116Z Generating XML reports...
2022-12-01T10:45:06.7135564Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104320.xml
2022-12-01T10:45:06.7135935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7136111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7136518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7136718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7136739Z 
2022-12-01T10:45:06.7136848Z Running tests...
2022-12-01T10:45:06.7137112Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7137426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7137688Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T10:45:06.7137708Z 
2022-12-01T10:45:06.7137974Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7138088Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7138108Z 
2022-12-01T10:45:06.7138214Z OK (skipped=1)
2022-12-01T10:45:06.7138233Z 
2022-12-01T10:45:06.7138344Z Generating XML reports...
2022-12-01T10:45:06.7138799Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104322.xml
2022-12-01T10:45:06.7139171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7139346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7139726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7139918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7139938Z 
2022-12-01T10:45:06.7140045Z Running tests...
2022-12-01T10:45:06.7140307Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7140603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7140992Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T10:45:06.7141057Z 
2022-12-01T10:45:06.7141327Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7141442Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7141461Z 
2022-12-01T10:45:06.7141570Z OK (skipped=1)
2022-12-01T10:45:06.7141590Z 
2022-12-01T10:45:06.7141713Z Generating XML reports...
2022-12-01T10:45:06.7142162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104324.xml
2022-12-01T10:45:06.7142536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7142713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7143072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7143268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7143288Z 
2022-12-01T10:45:06.7143401Z Running tests...
2022-12-01T10:45:06.7143665Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7143976Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7144241Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T10:45:06.7144261Z 
2022-12-01T10:45:06.7144523Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7144636Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7144656Z 
2022-12-01T10:45:06.7144763Z OK (skipped=1)
2022-12-01T10:45:06.7144782Z 
2022-12-01T10:45:06.7144887Z Generating XML reports...
2022-12-01T10:45:06.7145330Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104326.xml
2022-12-01T10:45:06.7145747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7145929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7146307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7146498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7146518Z 
2022-12-01T10:45:06.7146629Z Running tests...
2022-12-01T10:45:06.7146893Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7147186Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7147471Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-12-01T10:45:06.7147491Z 
2022-12-01T10:45:06.7147750Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7147867Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7147889Z 
2022-12-01T10:45:06.7147997Z OK (skipped=1)
2022-12-01T10:45:06.7148016Z 
2022-12-01T10:45:06.7148140Z Generating XML reports...
2022-12-01T10:45:06.7148586Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104328.xml
2022-12-01T10:45:06.7148956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7149133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7149717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7149917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7149938Z 
2022-12-01T10:45:06.7150047Z Running tests...
2022-12-01T10:45:06.7150319Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7150721Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7151035Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-12-01T10:45:06.7151055Z 
2022-12-01T10:45:06.7151316Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7151432Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7151451Z 
2022-12-01T10:45:06.7151560Z OK (skipped=1)
2022-12-01T10:45:06.7151579Z 
2022-12-01T10:45:06.7151684Z Generating XML reports...
2022-12-01T10:45:06.7152134Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104330.xml
2022-12-01T10:45:06.7152505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7152685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7153067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7153259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7153278Z 
2022-12-01T10:45:06.7153389Z Running tests...
2022-12-01T10:45:06.7153657Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7153971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7154321Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-12-01T10:45:06.7154379Z 
2022-12-01T10:45:06.7154841Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7155003Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7155023Z 
2022-12-01T10:45:06.7155131Z OK (skipped=1)
2022-12-01T10:45:06.7155156Z 
2022-12-01T10:45:06.7155280Z Generating XML reports...
2022-12-01T10:45:06.7155810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104332.xml
2022-12-01T10:45:06.7156202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7156380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7156763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7156937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7156956Z 
2022-12-01T10:45:06.7157064Z Running tests...
2022-12-01T10:45:06.7157325Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7157640Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7157940Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T10:45:06.7157959Z 
2022-12-01T10:45:06.7158223Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7158338Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7158356Z 
2022-12-01T10:45:06.7158466Z OK (skipped=1)
2022-12-01T10:45:06.7158485Z 
2022-12-01T10:45:06.7158609Z Generating XML reports...
2022-12-01T10:45:06.7159040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104334.xml
2022-12-01T10:45:06.7159411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7159588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7159973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7160222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7160241Z 
2022-12-01T10:45:06.7160351Z Running tests...
2022-12-01T10:45:06.7160619Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7160938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7161172Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7161394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55654
2022-12-01T10:45:06.7161611Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55655
2022-12-01T10:45:06.7161984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7162162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7162554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7162747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7163119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7163298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7163657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7163851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7164098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7164343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7164790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7165203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7165438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7165669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7165755Z ok (5.535s)
2022-12-01T10:45:06.7165793Z 
2022-12-01T10:45:06.7166039Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7166155Z Ran 1 test in 5.535s
2022-12-01T10:45:06.7166175Z 
2022-12-01T10:45:06.7166269Z OK
2022-12-01T10:45:06.7166288Z 
2022-12-01T10:45:06.7166414Z Generating XML reports...
2022-12-01T10:45:06.7166869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104336.xml
2022-12-01T10:45:06.7167248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7167425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7167806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7167987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7168006Z 
2022-12-01T10:45:06.7168117Z Running tests...
2022-12-01T10:45:06.7168382Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7168698Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7168982Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7169201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55883
2022-12-01T10:45:06.7169477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55884
2022-12-01T10:45:06.7169852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7170011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7170393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7170586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7170961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7171141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7171518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7171713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7171964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7172209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7172592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7172990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7173224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7173457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7173799Z STAGE:2022-12-01 10:43:49 55883:55883 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7174178Z STAGE:2022-12-01 10:43:49 55884:55884 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7174527Z STAGE:2022-12-01 10:43:49 55884:55884 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7174852Z STAGE:2022-12-01 10:43:49 55883:55883 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7174955Z ok (6.009s)
2022-12-01T10:45:06.7174974Z 
2022-12-01T10:45:06.7175221Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7175338Z Ran 1 test in 6.010s
2022-12-01T10:45:06.7175357Z 
2022-12-01T10:45:06.7175452Z OK
2022-12-01T10:45:06.7175473Z 
2022-12-01T10:45:06.7175598Z Generating XML reports...
2022-12-01T10:45:06.7176053Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104344.xml
2022-12-01T10:45:06.7176421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7176606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7176988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7177162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7177201Z 
2022-12-01T10:45:06.7177292Z Running tests...
2022-12-01T10:45:06.7177559Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7177871Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7178152Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7178371Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56074
2022-12-01T10:45:06.7178591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56075
2022-12-01T10:45:06.7179042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7179222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7179635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7179831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7180202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7180377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7180754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7180945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7181194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7181446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7181829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7182228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7182462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7182694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7183033Z STAGE:2022-12-01 10:43:57 56075:56075 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7183361Z STAGE:2022-12-01 10:43:57 56074:56074 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:45:06.7183749Z STAGE:2022-12-01 10:43:57 56075:56075 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7184094Z STAGE:2022-12-01 10:43:57 56074:56074 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:45:06.7184198Z ok (6.043s)
2022-12-01T10:45:06.7184217Z 
2022-12-01T10:45:06.7184462Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7184581Z Ran 1 test in 6.044s
2022-12-01T10:45:06.7184600Z 
2022-12-01T10:45:06.7184696Z OK
2022-12-01T10:45:06.7184715Z 
2022-12-01T10:45:06.7184841Z Generating XML reports...
2022-12-01T10:45:06.7185293Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104352.xml
2022-12-01T10:45:06.7185669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7185847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7186236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7186429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7186448Z 
2022-12-01T10:45:06.7186537Z Running tests...
2022-12-01T10:45:06.7186808Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7187128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7187419Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T10:45:06.7187438Z 
2022-12-01T10:45:06.7187703Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7187816Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7187836Z 
2022-12-01T10:45:06.7187945Z OK (skipped=1)
2022-12-01T10:45:06.7187963Z 
2022-12-01T10:45:06.7188089Z Generating XML reports...
2022-12-01T10:45:06.7188620Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104400.xml
2022-12-01T10:45:06.7188975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7189152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7189846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7190047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7190066Z 
2022-12-01T10:45:06.7190177Z Running tests...
2022-12-01T10:45:06.7190449Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7190764Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7191039Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T10:45:06.7191067Z 
2022-12-01T10:45:06.7191332Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7191426Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7191445Z 
2022-12-01T10:45:06.7191552Z OK (skipped=1)
2022-12-01T10:45:06.7191572Z 
2022-12-01T10:45:06.7191696Z Generating XML reports...
2022-12-01T10:45:06.7192150Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104402.xml
2022-12-01T10:45:06.7192522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7192701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7193084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7193276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7193299Z 
2022-12-01T10:45:06.7193463Z Running tests...
2022-12-01T10:45:06.7193743Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7194062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7194361Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T10:45:06.7194380Z 
2022-12-01T10:45:06.7194644Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7194757Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7194777Z 
2022-12-01T10:45:06.7194887Z OK (skipped=1)
2022-12-01T10:45:06.7194906Z 
2022-12-01T10:45:06.7195031Z Generating XML reports...
2022-12-01T10:45:06.7195477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104404.xml
2022-12-01T10:45:06.7195842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7196018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7196400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7196591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7196611Z 
2022-12-01T10:45:06.7196721Z Running tests...
2022-12-01T10:45:06.7196983Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7197303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7197600Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T10:45:06.7197619Z 
2022-12-01T10:45:06.7197881Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7198080Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7198102Z 
2022-12-01T10:45:06.7198210Z OK (skipped=1)
2022-12-01T10:45:06.7198228Z 
2022-12-01T10:45:06.7198354Z Generating XML reports...
2022-12-01T10:45:06.7198802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104406.xml
2022-12-01T10:45:06.7199177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7199355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7199738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7199931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7199951Z 
2022-12-01T10:45:06.7200060Z Running tests...
2022-12-01T10:45:06.7200305Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7200627Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7200914Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T10:45:06.7200933Z 
2022-12-01T10:45:06.7201195Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7201308Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7201327Z 
2022-12-01T10:45:06.7201435Z OK (skipped=1)
2022-12-01T10:45:06.7201454Z 
2022-12-01T10:45:06.7201581Z Generating XML reports...
2022-12-01T10:45:06.7202029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104408.xml
2022-12-01T10:45:06.7202380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7202557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7202989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7203187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7203206Z 
2022-12-01T10:45:06.7203316Z Running tests...
2022-12-01T10:45:06.7203579Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7203893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7204187Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T10:45:06.7204207Z 
2022-12-01T10:45:06.7204469Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7204562Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7204580Z 
2022-12-01T10:45:06.7204688Z OK (skipped=1)
2022-12-01T10:45:06.7204712Z 
2022-12-01T10:45:06.7204839Z Generating XML reports...
2022-12-01T10:45:06.7205287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104410.xml
2022-12-01T10:45:06.7205655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7205833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7206214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7206407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7206426Z 
2022-12-01T10:45:06.7206535Z Running tests...
2022-12-01T10:45:06.7206778Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7207091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7207416Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7207640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56763
2022-12-01T10:45:06.7207856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56764
2022-12-01T10:45:06.7208232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7208416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7208802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7208972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7209334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7209510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7209899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7210092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7210340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7210589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7210994Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7211395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7211610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7211840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7211947Z ok (6.912s)
2022-12-01T10:45:06.7212011Z 
2022-12-01T10:45:06.7212285Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7212399Z Ran 1 test in 6.912s
2022-12-01T10:45:06.7212418Z 
2022-12-01T10:45:06.7212513Z OK
2022-12-01T10:45:06.7212533Z 
2022-12-01T10:45:06.7212658Z Generating XML reports...
2022-12-01T10:45:06.7213110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104412.xml
2022-12-01T10:45:06.7213463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7213643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7214028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7214219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7214242Z 
2022-12-01T10:45:06.7214355Z Running tests...
2022-12-01T10:45:06.7214621Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7214940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7215216Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-12-01T10:45:06.7215236Z 
2022-12-01T10:45:06.7215499Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7215593Z Ran 1 test in 0.002s
2022-12-01T10:45:06.7215612Z 
2022-12-01T10:45:06.7215720Z OK (skipped=1)
2022-12-01T10:45:06.7215740Z 
2022-12-01T10:45:06.7215863Z Generating XML reports...
2022-12-01T10:45:06.7216312Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104421.xml
2022-12-01T10:45:06.7216740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7216916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7217298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7217492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7217511Z 
2022-12-01T10:45:06.7217622Z Running tests...
2022-12-01T10:45:06.7217868Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7218184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7218436Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7218657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57122
2022-12-01T10:45:06.7218881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57123
2022-12-01T10:45:06.7219259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7219436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7219819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7219996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7220361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7220536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7220910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7221100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7221393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7221645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7222047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7222446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7222659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7222889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7223152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyrnoqw8y
2022-12-01T10:45:06.7223428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyrnoqw8y/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7223688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5aqvmlnm
2022-12-01T10:45:06.7223957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5aqvmlnm/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7224060Z ok (6.633s)
2022-12-01T10:45:06.7224080Z 
2022-12-01T10:45:06.7224350Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7224445Z Ran 1 test in 6.633s
2022-12-01T10:45:06.7224484Z 
2022-12-01T10:45:06.7224561Z OK
2022-12-01T10:45:06.7224580Z 
2022-12-01T10:45:06.7224704Z Generating XML reports...
2022-12-01T10:45:06.7225155Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104423.xml
2022-12-01T10:45:06.7225528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7225753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7226138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7226331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7226351Z 
2022-12-01T10:45:06.7226464Z Running tests...
2022-12-01T10:45:06.7226899Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7227303Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7227598Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7227818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57389
2022-12-01T10:45:06.7228036Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57390
2022-12-01T10:45:06.7228408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7228593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7228974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7229167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7229737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7229921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7230308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7230503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7230751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7231083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7231498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7231897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7232110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7232340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7232599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjy5cy9b5
2022-12-01T10:45:06.7232868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjy5cy9b5/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7233124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_v_2q44k
2022-12-01T10:45:06.7233396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_v_2q44k/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7234178Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.7234956Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:45:06.7235127Z ok (7.042s)
2022-12-01T10:45:06.7235149Z 
2022-12-01T10:45:06.7235421Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7235537Z Ran 1 test in 7.043s
2022-12-01T10:45:06.7235557Z 
2022-12-01T10:45:06.7235653Z OK
2022-12-01T10:45:06.7235672Z 
2022-12-01T10:45:06.7235778Z Generating XML reports...
2022-12-01T10:45:06.7236236Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104432.xml
2022-12-01T10:45:06.7236607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7236787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7237176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7237371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7237390Z 
2022-12-01T10:45:06.7237499Z Running tests...
2022-12-01T10:45:06.7237765Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7238086Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7238355Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7238576Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57692
2022-12-01T10:45:06.7238795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57693
2022-12-01T10:45:06.7239168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7239397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7239787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7239983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7240347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7240525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7240877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7241068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7241316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7241568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7241972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7242368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7242599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7242829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7243052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.7243294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.7243695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.7244151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.7244392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.7244629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.7245017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.7245412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.7245677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt1v7ja6
2022-12-01T10:45:06.7245951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt1v7ja6/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7246195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppd6wx5mk
2022-12-01T10:45:06.7246469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppd6wx5mk/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7246573Z ok (10.828s)
2022-12-01T10:45:06.7246593Z 
2022-12-01T10:45:06.7246860Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7246977Z Ran 1 test in 10.828s
2022-12-01T10:45:06.7246997Z 
2022-12-01T10:45:06.7247092Z OK
2022-12-01T10:45:06.7247111Z 
2022-12-01T10:45:06.7247237Z Generating XML reports...
2022-12-01T10:45:06.7247693Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104441.xml
2022-12-01T10:45:06.7248049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7248229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7248663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7248863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7248882Z 
2022-12-01T10:45:06.7248992Z Running tests...
2022-12-01T10:45:06.7249256Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7249573Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T10:45:06.7249862Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:45:06.7250084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58033
2022-12-01T10:45:06.7250287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58034
2022-12-01T10:45:06.7250657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7250842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7251225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7251417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7251783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:45:06.7251960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:45:06.7252337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:45:06.7252509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:45:06.7252758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:45:06.7253069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:45:06.7253476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7253874Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:45:06.7254106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:45:06.7254334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:45:06.7254573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:45:06.7254813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:45:06.7255193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.7255596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:45:06.7255837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:45:06.7256077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:45:06.7256466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.7256861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:45:06.7257121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lvprhly
2022-12-01T10:45:06.7257397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lvprhly/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7257655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22anrixd
2022-12-01T10:45:06.7257947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22anrixd/_remote_module_non_scriptable.py
2022-12-01T10:45:06.7258064Z ok (10.998s)
2022-12-01T10:45:06.7258083Z 
2022-12-01T10:45:06.7258351Z ----------------------------------------------------------------------
2022-12-01T10:45:06.7258468Z Ran 1 test in 10.998s
2022-12-01T10:45:06.7258487Z 
2022-12-01T10:45:06.7258582Z OK
2022-12-01T10:45:06.7258601Z 
2022-12-01T10:45:06.7258726Z Generating XML reports...
2022-12-01T10:45:06.7259179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104454.xml
2022-12-01T10:45:06.7259199Z 
2022-12-01T10:45:06.7259648Z ##[endgroup]
2022-12-01T10:45:06.7260099Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_a2b6lcu2)
2022-12-01T10:45:06.7260124Z 
2022-12-01T10:48:19.7851568Z 
2022-12-01T10:48:19.7852108Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:48:19.7855329Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_z7t7n6hj)
2022-12-01T10:48:19.7863996Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:48:19.7907085Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T10:48:19.7955516Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7958234Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7959037Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7959810Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7960629Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7961445Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7962271Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7963197Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7964151Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7964749Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7965399Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7966049Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7966974Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7967497Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7967979Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7968456Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7969219Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7970036Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7970756Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7971599Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7972358Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7973250Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7973988Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7974730Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7975541Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7976328Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7977087Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7977873Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7978583Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7979262Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7979968Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7980687Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7981415Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7981916Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7982349Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7982765Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7983432Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7984390Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7985203Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7985878Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7986306Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7986751Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7987182Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7987637Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7988293Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7988729Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7989153Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7990222Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7990740Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7991161Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7991609Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7992019Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7992433Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7992973Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7993527Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7994282Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7994930Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7995848Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7996567Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7997272Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7997970Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7998714Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7999312Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.7999797Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8000515Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8001266Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8002041Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8002716Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8003425Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8004125Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8004681Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8005427Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8006174Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8006920Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8007625Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8008237Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8008887Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8009268Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8009666Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8010079Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8010556Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8010979Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8011383Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8011813Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8012239Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8012685Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8013139Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8013573Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8014026Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8014481Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8015032Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8015521Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8016158Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8016703Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8017189Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8017777Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8018316Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8018848Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8019893Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8020390Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8020855Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8021257Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8021752Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8022217Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8022629Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8023129Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8023620Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8024095Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8024522Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8025005Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8025504Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8025930Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8026469Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8027013Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8027452Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8028010Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8028498Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8029044Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8030291Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8030835Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8031345Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8031862Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8032358Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8032889Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8033400Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8033820Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8034678Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8035277Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8035761Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8063157Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8063609Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8064050Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8064504Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8064980Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8065424Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8065837Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8066284Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8066692Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8067051Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8067443Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8067863Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8068454Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8068898Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8069345Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8070644Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8071054Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8071521Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8072033Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8072589Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8073213Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8073846Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8074460Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8075073Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8075670Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8076279Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8077128Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8077708Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8078229Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8078681Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8079073Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8079455Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8079868Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8080267Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8080698Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8081124Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8081596Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8082070Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8082469Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8082863Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8083281Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8083706Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8084144Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8084560Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8084975Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8085387Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8085922Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8086359Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8086756Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8087171Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8087603Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8088008Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8088416Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8088883Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8089318Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8089684Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8090093Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8090531Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8090956Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8091327Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8091697Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8092073Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8092436Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8092822Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8093200Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8093571Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8093964Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8094336Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8094682Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8095126Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8095544Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8095942Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8096300Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8096657Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8097035Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8097430Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8097853Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8098315Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8098777Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8099186Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8099618Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8100062Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8100486Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8100914Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8101340Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8101757Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8102151Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8102569Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8102968Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8103361Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8103907Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8104408Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8104884Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8105323Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8105794Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8106253Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8106669Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8107098Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8107531Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8107977Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8108412Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8108892Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8109799Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8110276Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8110697Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8111093Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8111499Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8111885Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8112272Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8112661Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8113120Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8113521Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8113893Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8114236Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8114614Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8114992Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8115368Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8115773Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8116149Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8116519Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8116891Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8117287Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8117655Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8118001Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8118378Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8118754Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8119119Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8119510Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8119891Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8120275Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8120626Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8121002Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8121421Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8121934Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8122366Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8122762Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8123148Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8123579Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8123992Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8124391Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8124787Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8125227Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8125640Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8126037Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8126447Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8126849Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8127215Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8127637Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8128088Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8128535Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8129249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8129707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8130285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8130816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8131040Z 
2022-12-01T10:48:19.8131149Z Running tests...
2022-12-01T10:48:19.8131558Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8132091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8132665Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8133228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1486
2022-12-01T10:48:19.8133679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1488
2022-12-01T10:48:19.8134284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8134723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8135301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8135769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8136323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8136770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8137338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8137800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8138233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8138735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8139477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8140169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8140678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8141153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8141682Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8142496Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8143160Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8143982Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8144632Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8145444Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8146090Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8146906Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8147558Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8148429Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8149086Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8150616Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8151282Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8152099Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8152745Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T10:48:19.8153570Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T10:48:19.8154055Z ok (7.243s)
2022-12-01T10:48:19.8154207Z 
2022-12-01T10:48:19.8154477Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8154793Z Ran 1 test in 7.243s
2022-12-01T10:48:19.8154954Z 
2022-12-01T10:48:19.8155050Z OK
2022-12-01T10:48:19.8155183Z 
2022-12-01T10:48:19.8155307Z Generating XML reports...
2022-12-01T10:48:19.8155899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102113.xml
2022-12-01T10:48:19.8156613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8157062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8157769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8158227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8158456Z 
2022-12-01T10:48:19.8158570Z Running tests...
2022-12-01T10:48:19.8158976Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8159489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8160023Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-12-01T10:48:19.8160333Z 
2022-12-01T10:48:19.8160594Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8160920Z Ran 1 test in 0.004s
2022-12-01T10:48:19.8161062Z 
2022-12-01T10:48:19.8161175Z OK (skipped=1)
2022-12-01T10:48:19.8161327Z 
2022-12-01T10:48:19.8161458Z Generating XML reports...
2022-12-01T10:48:19.8162062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102122.xml
2022-12-01T10:48:19.8162758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8163208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8163787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8164256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8164484Z 
2022-12-01T10:48:19.8164575Z Running tests...
2022-12-01T10:48:19.8164980Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8165517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8166027Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8166577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1922
2022-12-01T10:48:19.8167042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1923
2022-12-01T10:48:19.8167650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8168079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8168652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8169120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8169695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8170124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8170707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8171169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8171601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8172262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8172797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8173446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8173949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8174421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8174833Z ok (4.077s)
2022-12-01T10:48:19.8174982Z 
2022-12-01T10:48:19.8175260Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8175571Z Ran 1 test in 4.077s
2022-12-01T10:48:19.8175732Z 
2022-12-01T10:48:19.8175826Z OK
2022-12-01T10:48:19.8175961Z 
2022-12-01T10:48:19.8176085Z Generating XML reports...
2022-12-01T10:48:19.8176673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102124.xml
2022-12-01T10:48:19.8177387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8177837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8178477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8178932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8179166Z 
2022-12-01T10:48:19.8179276Z Running tests...
2022-12-01T10:48:19.8179687Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8180202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8180734Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8181788Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.672s)
2022-12-01T10:48:19.8182315Z 
2022-12-01T10:48:19.8182580Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8182906Z Ran 1 test in 1.672s
2022-12-01T10:48:19.8183049Z 
2022-12-01T10:48:19.8183163Z OK (skipped=1)
2022-12-01T10:48:19.8183316Z 
2022-12-01T10:48:19.8183443Z Generating XML reports...
2022-12-01T10:48:19.8184102Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml
2022-12-01T10:48:19.8184812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8185266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8185840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8186310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8186538Z 
2022-12-01T10:48:19.8186628Z Running tests...
2022-12-01T10:48:19.8187032Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8187560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8188089Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8188602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2373
2022-12-01T10:48:19.8189050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2374
2022-12-01T10:48:19.8190118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8190554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8191136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8191604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8192180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8192723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8193302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8193765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8194200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8194699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8195357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8196049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8196556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8197037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8197539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp68vxuuby
2022-12-01T10:48:19.8198081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp68vxuuby/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8198597Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpih8zve8l
2022-12-01T10:48:19.8199132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpih8zve8l/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8199647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8200115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8200464Z ok (3.831s)
2022-12-01T10:48:19.8200612Z 
2022-12-01T10:48:19.8200887Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8201218Z Ran 1 test in 3.831s
2022-12-01T10:48:19.8201361Z 
2022-12-01T10:48:19.8201455Z OK
2022-12-01T10:48:19.8201665Z 
2022-12-01T10:48:19.8201800Z Generating XML reports...
2022-12-01T10:48:19.8202418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml
2022-12-01T10:48:19.8203120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8203570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8204146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8204613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8204825Z 
2022-12-01T10:48:19.8204934Z Running tests...
2022-12-01T10:48:19.8205337Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8205877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8206413Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8206940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2701
2022-12-01T10:48:19.8207390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2702
2022-12-01T10:48:19.8207995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8208426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8209003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8209471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8210033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8210553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8211132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8211599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8212031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8212528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8213185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8213873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8214380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8214857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8215355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_k03a3a
2022-12-01T10:48:19.8215860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hnjp5zz
2022-12-01T10:48:19.8216398Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_k03a3a/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8216937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hnjp5zz/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8217455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8217923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8218275Z ok (3.985s)
2022-12-01T10:48:19.8218431Z 
2022-12-01T10:48:19.8218707Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8219072Z Ran 1 test in 3.986s
2022-12-01T10:48:19.8219247Z 
2022-12-01T10:48:19.8219342Z OK
2022-12-01T10:48:19.8219478Z 
2022-12-01T10:48:19.8219601Z Generating XML reports...
2022-12-01T10:48:19.8220211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102141.xml
2022-12-01T10:48:19.8220906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8221359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8221934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8222408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8222619Z 
2022-12-01T10:48:19.8222727Z Running tests...
2022-12-01T10:48:19.8223134Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8223672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8224202Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8224728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2945
2022-12-01T10:48:19.8225180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2946
2022-12-01T10:48:19.8225786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8226216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8226791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8227258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8227887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8228330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8228901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8229835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8230515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8231015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8231684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8232378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8232892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8233462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8234141Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbb_plz50
2022-12-01T10:48:19.8234668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbb_plz50/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8235198Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt0py9r7
2022-12-01T10:48:19.8235731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt0py9r7/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8236242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8236710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8237303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8237800Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8238253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8238738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8239217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8239688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8240144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8240615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8241086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8241548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8241902Z ok (6.947s)
2022-12-01T10:48:19.8242054Z 
2022-12-01T10:48:19.8242346Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8242676Z Ran 1 test in 6.947s
2022-12-01T10:48:19.8242819Z 
2022-12-01T10:48:19.8242913Z OK
2022-12-01T10:48:19.8243048Z 
2022-12-01T10:48:19.8243174Z Generating XML reports...
2022-12-01T10:48:19.8243782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102147.xml
2022-12-01T10:48:19.8244482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8244930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8245505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8246068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8246301Z 
2022-12-01T10:48:19.8246392Z Running tests...
2022-12-01T10:48:19.8246800Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8247328Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8247872Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8248416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3296
2022-12-01T10:48:19.8248863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3297
2022-12-01T10:48:19.8249465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8249902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8250485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8250957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8251531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8251956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8252524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8252992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8253425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8254070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8254930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8255638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8256144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8256617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8257129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2cewn69b
2022-12-01T10:48:19.8257667Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2cewn69b/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8258178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshtas81f
2022-12-01T10:48:19.8258714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshtas81f/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8259236Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8259701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8260180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8260665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8261014Z ok (5.558s)
2022-12-01T10:48:19.8261146Z 
2022-12-01T10:48:19.8261422Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8261754Z Ran 1 test in 5.558s
2022-12-01T10:48:19.8261917Z 
2022-12-01T10:48:19.8262013Z OK
2022-12-01T10:48:19.8262149Z 
2022-12-01T10:48:19.8262257Z Generating XML reports...
2022-12-01T10:48:19.8262866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102156.xml
2022-12-01T10:48:19.8263663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8264117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8264677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8265189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8265420Z 
2022-12-01T10:48:19.8265533Z Running tests...
2022-12-01T10:48:19.8265926Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8266459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8267282Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8267840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3604
2022-12-01T10:48:19.8268284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3605
2022-12-01T10:48:19.8268904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8269732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8270558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8271013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8271589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8272034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8272586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8273155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8273626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8274133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8274784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8275474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8275999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8276473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8276956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9dc8ujax
2022-12-01T10:48:19.8277508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9dc8ujax/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8278085Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ws_y9st
2022-12-01T10:48:19.8278605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ws_y9st/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8279119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8279602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8280142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8280855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8281210Z ok (5.811s)
2022-12-01T10:48:19.8281361Z 
2022-12-01T10:48:19.8281647Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8282068Z Ran 1 test in 5.811s
2022-12-01T10:48:19.8282235Z 
2022-12-01T10:48:19.8282340Z OK
2022-12-01T10:48:19.8282476Z 
2022-12-01T10:48:19.8282604Z Generating XML reports...
2022-12-01T10:48:19.8283220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102204.xml
2022-12-01T10:48:19.8284167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8284716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8285297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8285751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8285984Z 
2022-12-01T10:48:19.8286097Z Running tests...
2022-12-01T10:48:19.8286505Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8287048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8287625Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8288192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3946
2022-12-01T10:48:19.8288642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3947
2022-12-01T10:48:19.8289230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8289684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8290259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8290725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8291359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8291823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8292396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8293096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8293583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8294088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8294750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8295422Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8295955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8296429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8296928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphv9cwke1
2022-12-01T10:48:19.8297450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphv9cwke1/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8297978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42mczzdu
2022-12-01T10:48:19.8298508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42mczzdu/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8299021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8299487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8299965Z ok (6.025s)
2022-12-01T10:48:19.8300234Z 
2022-12-01T10:48:19.8300722Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8301296Z Ran 1 test in 6.026s
2022-12-01T10:48:19.8301603Z 
2022-12-01T10:48:19.8301763Z OK
2022-12-01T10:48:19.8302029Z 
2022-12-01T10:48:19.8302272Z Generating XML reports...
2022-12-01T10:48:19.8303514Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102212.xml
2022-12-01T10:48:19.8305008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8305974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8307186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8308101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8308561Z 
2022-12-01T10:48:19.8308798Z Running tests...
2022-12-01T10:48:19.8309987Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8311073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8312204Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8313391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4296
2022-12-01T10:48:19.8314307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4297
2022-12-01T10:48:19.8315540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8316391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8317568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8318707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8319899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8320793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8321971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8322899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8323804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8324798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8326148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8327538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8328610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8329573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8330565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwv0ogh5
2022-12-01T10:48:19.8331601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwv0ogh5/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8332667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq50st0hh
2022-12-01T10:48:19.8333747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq50st0hh/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8334736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8335893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8336977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8337980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8338650Z ok (6.583s)
2022-12-01T10:48:19.8338938Z 
2022-12-01T10:48:19.8339506Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8340144Z Ran 1 test in 6.583s
2022-12-01T10:48:19.8340453Z 
2022-12-01T10:48:19.8340602Z OK
2022-12-01T10:48:19.8340861Z 
2022-12-01T10:48:19.8341103Z Generating XML reports...
2022-12-01T10:48:19.8342337Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102221.xml
2022-12-01T10:48:19.8343795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8344689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8345870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8346820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8347279Z 
2022-12-01T10:48:19.8347482Z Running tests...
2022-12-01T10:48:19.8348249Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8349308Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8350826Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8351901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4647
2022-12-01T10:48:19.8352822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4648
2022-12-01T10:48:19.8354271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8355186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8356351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8357302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8358463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8359307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8360487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8361431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8362337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8363337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8364725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8366131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8367193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8368099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8369104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdhwsz6ps
2022-12-01T10:48:19.8370180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdhwsz6ps/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8371207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69o8vh79
2022-12-01T10:48:19.8372481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69o8vh79/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8373477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8374459Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8375382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8376342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8377047Z ok (6.282s)
2022-12-01T10:48:19.8377334Z 
2022-12-01T10:48:19.8377873Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8378569Z Ran 1 test in 6.283s
2022-12-01T10:48:19.8378880Z 
2022-12-01T10:48:19.8379054Z OK
2022-12-01T10:48:19.8379306Z 
2022-12-01T10:48:19.8379550Z Generating XML reports...
2022-12-01T10:48:19.8380771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102229.xml
2022-12-01T10:48:19.8382247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8383146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8384296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8385271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8385730Z 
2022-12-01T10:48:19.8385942Z Running tests...
2022-12-01T10:48:19.8386756Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8387800Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8388966Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8390629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4991
2022-12-01T10:48:19.8391556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4992
2022-12-01T10:48:19.8392785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8393686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8394880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8395776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8396973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8397870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8399066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8399994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8400908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8401907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8403359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8404774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8405820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8406768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8408015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw6djpzm
2022-12-01T10:48:19.8409103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw6djpzm/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8410177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2m_u7yw
2022-12-01T10:48:19.8411259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2m_u7yw/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8412294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8413267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8414253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8415198Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8415923Z ok (5.680s)
2022-12-01T10:48:19.8416213Z 
2022-12-01T10:48:19.8416775Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8417396Z Ran 1 test in 5.680s
2022-12-01T10:48:19.8417675Z 
2022-12-01T10:48:19.8417848Z OK
2022-12-01T10:48:19.8418104Z 
2022-12-01T10:48:19.8418344Z Generating XML reports...
2022-12-01T10:48:19.8419565Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102238.xml
2022-12-01T10:48:19.8421017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8421928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8423129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8424093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8424531Z 
2022-12-01T10:48:19.8424743Z Running tests...
2022-12-01T10:48:19.8425685Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8426787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8427858Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8430337Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.678s)
2022-12-01T10:48:19.8431377Z 
2022-12-01T10:48:19.8431924Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8432579Z Ran 1 test in 1.679s
2022-12-01T10:48:19.8432899Z 
2022-12-01T10:48:19.8433130Z OK (skipped=1)
2022-12-01T10:48:19.8433408Z 
2022-12-01T10:48:19.8433653Z Generating XML reports...
2022-12-01T10:48:19.8434880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102246.xml
2022-12-01T10:48:19.8436341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8437224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8438420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8439345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8439797Z 
2022-12-01T10:48:19.8439999Z Running tests...
2022-12-01T10:48:19.8440762Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8441819Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8443110Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8444148Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5360
2022-12-01T10:48:19.8445078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5361
2022-12-01T10:48:19.8446345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8447269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8448430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8449392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8450602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8451506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8452716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8453670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8454577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8455543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8456902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8458291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8459342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8460468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8461175Z ok (4.113s)
2022-12-01T10:48:19.8461472Z 
2022-12-01T10:48:19.8462019Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8462625Z Ran 1 test in 4.113s
2022-12-01T10:48:19.8462930Z 
2022-12-01T10:48:19.8463094Z OK
2022-12-01T10:48:19.8463340Z 
2022-12-01T10:48:19.8463584Z Generating XML reports...
2022-12-01T10:48:19.8464805Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102250.xml
2022-12-01T10:48:19.8466233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8467137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8468316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8469236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8470100Z 
2022-12-01T10:48:19.8470326Z Running tests...
2022-12-01T10:48:19.8471146Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8472208Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8473298Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8475456Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2022-12-01T10:48:19.8476474Z 
2022-12-01T10:48:19.8477019Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8477871Z Ran 1 test in 1.677s
2022-12-01T10:48:19.8478229Z 
2022-12-01T10:48:19.8478425Z OK (skipped=1)
2022-12-01T10:48:19.8478718Z 
2022-12-01T10:48:19.8478957Z Generating XML reports...
2022-12-01T10:48:19.8480193Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102256.xml
2022-12-01T10:48:19.8481646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8482523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8483683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8484604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8485054Z 
2022-12-01T10:48:19.8485241Z Running tests...
2022-12-01T10:48:19.8486046Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8487141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8488190Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8489174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5810
2022-12-01T10:48:19.8490085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5811
2022-12-01T10:48:19.8491387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8492260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8493411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8494355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8495558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8496612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8497809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8498752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8499626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8500638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8501971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8503386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8504460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8505397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8506083Z ok (5.566s)
2022-12-01T10:48:19.8506374Z 
2022-12-01T10:48:19.8506926Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8507531Z Ran 1 test in 5.566s
2022-12-01T10:48:19.8507845Z 
2022-12-01T10:48:19.8508023Z OK
2022-12-01T10:48:19.8508274Z 
2022-12-01T10:48:19.8508509Z Generating XML reports...
2022-12-01T10:48:19.8510070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102300.xml
2022-12-01T10:48:19.8511602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8512499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8513678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8514809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8515268Z 
2022-12-01T10:48:19.8515479Z Running tests...
2022-12-01T10:48:19.8516302Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8517361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8518398Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T10:48:19.8518975Z 
2022-12-01T10:48:19.8519496Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8520117Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8520412Z 
2022-12-01T10:48:19.8520625Z OK (skipped=1)
2022-12-01T10:48:19.8520919Z 
2022-12-01T10:48:19.8521163Z Generating XML reports...
2022-12-01T10:48:19.8522388Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102308.xml
2022-12-01T10:48:19.8523836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8524747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8525919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8526866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8527328Z 
2022-12-01T10:48:19.8527528Z Running tests...
2022-12-01T10:48:19.8528331Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8529393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8530284Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8531318Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8532481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6209
2022-12-01T10:48:19.8533411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6210
2022-12-01T10:48:19.8534683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8535603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8536777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8537846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8538993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8539894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8541088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8542018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8542922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8543922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8545291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8546705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8547766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8548818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7ppzd5v
2022-12-01T10:48:19.8550436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7ppzd5v/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8551452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8552430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4349fqm
2022-12-01T10:48:19.8553522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4349fqm/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8554243Z ok (4.114s)
2022-12-01T10:48:19.8554528Z 
2022-12-01T10:48:19.8555084Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8555722Z Ran 1 test in 4.115s
2022-12-01T10:48:19.8556042Z 
2022-12-01T10:48:19.8556217Z OK
2022-12-01T10:48:19.8556460Z 
2022-12-01T10:48:19.8556702Z Generating XML reports...
2022-12-01T10:48:19.8557940Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102310.xml
2022-12-01T10:48:19.8559416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8560291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8561487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8562442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8562915Z 
2022-12-01T10:48:19.8563127Z Running tests...
2022-12-01T10:48:19.8563930Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8565010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8565975Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8566951Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8568118Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6510
2022-12-01T10:48:19.8569059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6511
2022-12-01T10:48:19.8570323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8571189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8572368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8573288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8574484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8575382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8576549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8577500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8578412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8579417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8580778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8582207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8583255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8584283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf75gh2n1
2022-12-01T10:48:19.8585633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf75gh2n1/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8586672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8587717Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ccr8xn0
2022-12-01T10:48:19.8588805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ccr8xn0/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8590180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8591135Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8591855Z ok (4.205s)
2022-12-01T10:48:19.8592156Z 
2022-12-01T10:48:19.8592720Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8593333Z Ran 1 test in 4.206s
2022-12-01T10:48:19.8593646Z 
2022-12-01T10:48:19.8593842Z OK
2022-12-01T10:48:19.8594108Z 
2022-12-01T10:48:19.8594340Z Generating XML reports...
2022-12-01T10:48:19.8595568Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102316.xml
2022-12-01T10:48:19.8596997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8597894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8599072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8599989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8600457Z 
2022-12-01T10:48:19.8600664Z Running tests...
2022-12-01T10:48:19.8601466Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8602506Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8603672Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8604764Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8605793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6796
2022-12-01T10:48:19.8606727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6797
2022-12-01T10:48:19.8607936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8608847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8610031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8610962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8612177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8613072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8614265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8615189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8616109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8617112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8618446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8619890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8621140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8622154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9goaq9br
2022-12-01T10:48:19.8623218Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9goaq9br/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8624229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8625227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedoz11ld
2022-12-01T10:48:19.8626296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedoz11ld/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8627321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8628319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.8629005Z ok (4.028s)
2022-12-01T10:48:19.8629313Z 
2022-12-01T10:48:19.8630181Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8630828Z Ran 1 test in 4.028s
2022-12-01T10:48:19.8631139Z 
2022-12-01T10:48:19.8631318Z OK
2022-12-01T10:48:19.8631582Z 
2022-12-01T10:48:19.8631828Z Generating XML reports...
2022-12-01T10:48:19.8633045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102323.xml
2022-12-01T10:48:19.8634525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8635433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8636682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8637635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8638087Z 
2022-12-01T10:48:19.8638311Z Running tests...
2022-12-01T10:48:19.8639303Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8640377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8641336Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T10:48:19.8642320Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8643271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7071
2022-12-01T10:48:19.8644180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7072
2022-12-01T10:48:19.8645428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8646350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8647497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8648456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8649652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8650540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8651715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8652672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8653587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8654566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8655911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8657494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8658522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8659505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbimo6yx
2022-12-01T10:48:19.8660598Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbimo6yx/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8661629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8662628Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgi9oa7hc
2022-12-01T10:48:19.8663714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgi9oa7hc/_remote_module_non_scriptable.py
2022-12-01T10:48:19.8664481Z ok (4.115s)
2022-12-01T10:48:19.8664769Z 
2022-12-01T10:48:19.8665343Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8665978Z Ran 1 test in 4.116s
2022-12-01T10:48:19.8666295Z 
2022-12-01T10:48:19.8666466Z OK
2022-12-01T10:48:19.8666720Z 
2022-12-01T10:48:19.8666956Z Generating XML reports...
2022-12-01T10:48:19.8668168Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102329.xml
2022-12-01T10:48:19.8670001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8670926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8672121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8673045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8673499Z 
2022-12-01T10:48:19.8673710Z Running tests...
2022-12-01T10:48:19.8674540Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8675756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8676781Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8677715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7336
2022-12-01T10:48:19.8678673Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7337
2022-12-01T10:48:19.8679907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8680831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8681976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8682944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8684092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8684992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8686165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8687087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8687970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8688973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8690333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8691744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8692985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8694165Z STAGE:2022-12-01 10:23:39 7337:7337 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8695102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8696252Z STAGE:2022-12-01 10:23:39 7336:7336 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8697439Z STAGE:2022-12-01 10:23:39 7336:7336 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8698600Z STAGE:2022-12-01 10:23:39 7337:7337 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8699747Z STAGE:2022-12-01 10:23:39 7336:7336 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8700877Z STAGE:2022-12-01 10:23:39 7337:7337 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8702050Z STAGE:2022-12-01 10:23:39 7336:7336 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8703243Z STAGE:2022-12-01 10:23:39 7337:7337 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8703897Z ok (4.034s)
2022-12-01T10:48:19.8704186Z 
2022-12-01T10:48:19.8704725Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8705362Z Ran 1 test in 4.035s
2022-12-01T10:48:19.8705677Z 
2022-12-01T10:48:19.8705829Z OK
2022-12-01T10:48:19.8706087Z 
2022-12-01T10:48:19.8706327Z Generating XML reports...
2022-12-01T10:48:19.8707548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102336.xml
2022-12-01T10:48:19.8708989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8710205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8711585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8712563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8713033Z 
2022-12-01T10:48:19.8713223Z Running tests...
2022-12-01T10:48:19.8714042Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8715092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8716145Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8717136Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7653
2022-12-01T10:48:19.8718047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7654
2022-12-01T10:48:19.8719306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8720241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8720979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8721334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8722111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8722497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8723277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8723652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8724119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8724799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8725616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8726435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8726904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8727367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8728044Z STAGE:2022-12-01 10:23:45 7654:7654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8728697Z STAGE:2022-12-01 10:23:45 7653:7653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8730207Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8730443Z   warnings.warn(
2022-12-01T10:48:19.8731965Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8732187Z   warnings.warn(
2022-12-01T10:48:19.8733254Z STAGE:2022-12-01 10:23:45 7653:7653 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:23:45 7654:7654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8733316Z 
2022-12-01T10:48:19.8733941Z STAGE:2022-12-01 10:23:45 7653:7653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8734613Z STAGE:2022-12-01 10:23:45 7654:7654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8735402Z STAGE:2022-12-01 10:23:45 7654:7654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8736084Z STAGE:2022-12-01 10:23:45 7653:7653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8736738Z STAGE:2022-12-01 10:23:45 7653:7653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8737392Z STAGE:2022-12-01 10:23:45 7654:7654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8738065Z STAGE:2022-12-01 10:23:45 7654:7654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8738748Z STAGE:2022-12-01 10:23:45 7653:7653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8738928Z ok (3.853s)
2022-12-01T10:48:19.8738981Z 
2022-12-01T10:48:19.8739484Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8739715Z Ran 1 test in 3.853s
2022-12-01T10:48:19.8739745Z 
2022-12-01T10:48:19.8739918Z OK
2022-12-01T10:48:19.8739952Z 
2022-12-01T10:48:19.8740194Z Generating XML reports...
2022-12-01T10:48:19.8741120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102342.xml
2022-12-01T10:48:19.8741882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8742223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8743003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8743360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8743389Z 
2022-12-01T10:48:19.8743600Z Running tests...
2022-12-01T10:48:19.8744123Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8744894Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8745468Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8745905Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7928
2022-12-01T10:48:19.8746332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7929
2022-12-01T10:48:19.8747100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8747424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8748190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8748573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8749337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8750014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8750802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8751181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8751668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8752155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8752952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8753772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8754235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8754853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8755352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.8755843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.8756671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8757477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8758139Z STAGE:2022-12-01 10:23:51 7928:7928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8758779Z STAGE:2022-12-01 10:23:51 7929:7929 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8760310Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8760537Z   warnings.warn(
2022-12-01T10:48:19.8762040Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8762265Z   warnings.warn(
2022-12-01T10:48:19.8762947Z STAGE:2022-12-01 10:23:51 7928:7928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8763609Z STAGE:2022-12-01 10:23:51 7929:7929 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8764656Z STAGE:2022-12-01 10:23:51 7928:7928 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:23:51 7929:7929 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8764843Z 
2022-12-01T10:48:19.8765530Z STAGE:2022-12-01 10:23:51 7928:7928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8766206Z STAGE:2022-12-01 10:23:51 7929:7929 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8766870Z STAGE:2022-12-01 10:23:51 7928:7928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8767507Z STAGE:2022-12-01 10:23:51 7929:7929 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8768175Z STAGE:2022-12-01 10:23:51 7928:7928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8768845Z STAGE:2022-12-01 10:23:51 7929:7929 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8769040Z ok (4.064s)
2022-12-01T10:48:19.8769083Z 
2022-12-01T10:48:19.8769604Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8769812Z Ran 1 test in 4.064s
2022-12-01T10:48:19.8769841Z 
2022-12-01T10:48:19.8770010Z OK
2022-12-01T10:48:19.8770037Z 
2022-12-01T10:48:19.8770276Z Generating XML reports...
2022-12-01T10:48:19.8771177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102348.xml
2022-12-01T10:48:19.8771940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8772278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8773051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8773431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8773460Z 
2022-12-01T10:48:19.8773673Z Running tests...
2022-12-01T10:48:19.8774314Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8774975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8775511Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8775930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8185
2022-12-01T10:48:19.8776358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8186
2022-12-01T10:48:19.8777121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8777458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8778316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8778697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8779478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8779829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8780569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8780944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8781429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8781912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8782734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8783550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8784175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8784629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8784938Z skip: Skipped due to small world size. (3.910s)
2022-12-01T10:48:19.8784970Z 
2022-12-01T10:48:19.8785481Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8785699Z Ran 1 test in 3.911s
2022-12-01T10:48:19.8785727Z 
2022-12-01T10:48:19.8785929Z OK (skipped=1)
2022-12-01T10:48:19.8785957Z 
2022-12-01T10:48:19.8786196Z Generating XML reports...
2022-12-01T10:48:19.8787121Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102354.xml
2022-12-01T10:48:19.8787880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8788243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8789012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8789761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8789797Z 
2022-12-01T10:48:19.8790002Z Running tests...
2022-12-01T10:48:19.8790540Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8791198Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8791762Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8792193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8524
2022-12-01T10:48:19.8792624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8525
2022-12-01T10:48:19.8793551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8793908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8794676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8795044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8795822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8796171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8796968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8797350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8797822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8798326Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8799153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8799939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8800402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8801074Z STAGE:2022-12-01 10:24:04 8525:8525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8802572Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8803003Z   warnings.warn(
2022-12-01T10:48:19.8803462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8804168Z STAGE:2022-12-01 10:24:04 8524:8524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8805662Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8805892Z   warnings.warn(
2022-12-01T10:48:19.8806554Z STAGE:2022-12-01 10:24:04 8524:8524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8807232Z STAGE:2022-12-01 10:24:04 8525:8525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8807890Z STAGE:2022-12-01 10:24:04 8524:8524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8808572Z STAGE:2022-12-01 10:24:04 8525:8525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8809245Z STAGE:2022-12-01 10:24:04 8524:8524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8809908Z STAGE:2022-12-01 10:24:04 8525:8525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8810555Z STAGE:2022-12-01 10:24:04 8524:8524 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8811224Z STAGE:2022-12-01 10:24:04 8525:8525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8811907Z STAGE:2022-12-01 10:24:04 8524:8524 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8812528Z STAGE:2022-12-01 10:24:04 8525:8525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8812729Z ok (4.162s)
2022-12-01T10:48:19.8812767Z 
2022-12-01T10:48:19.8813292Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8813643Z Ran 1 test in 4.162s
2022-12-01T10:48:19.8813676Z 
2022-12-01T10:48:19.8813865Z OK
2022-12-01T10:48:19.8813891Z 
2022-12-01T10:48:19.8814144Z Generating XML reports...
2022-12-01T10:48:19.8815074Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102400.xml
2022-12-01T10:48:19.8815819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8816173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8816927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8817302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8817335Z 
2022-12-01T10:48:19.8817561Z Running tests...
2022-12-01T10:48:19.8818115Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8818754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8819321Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8819752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8794
2022-12-01T10:48:19.8820190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8795
2022-12-01T10:48:19.8820928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8821279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8822068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8822445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8823322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8823660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8824428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8824815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8825278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8825721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8826503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8827287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8827765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8828221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8828894Z STAGE:2022-12-01 10:24:10 8794:8794 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8829862Z STAGE:2022-12-01 10:24:10 8795:8795 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8831382Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8831608Z   warnings.warn(
2022-12-01T10:48:19.8833254Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.8833510Z   warnings.warn(
2022-12-01T10:48:19.8834602Z STAGE:2022-12-01 10:24:10 8795:8795 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:10 8794:8794 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8834636Z 
2022-12-01T10:48:19.8834838Z ok (4.067s)
2022-12-01T10:48:19.8834866Z 
2022-12-01T10:48:19.8835383Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8835611Z Ran 1 test in 4.067s
2022-12-01T10:48:19.8835643Z 
2022-12-01T10:48:19.8835825Z OK
2022-12-01T10:48:19.8835851Z 
2022-12-01T10:48:19.8836098Z Generating XML reports...
2022-12-01T10:48:19.8837030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102407.xml
2022-12-01T10:48:19.8837814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8838127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8838887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8839283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8839316Z 
2022-12-01T10:48:19.8839524Z Running tests...
2022-12-01T10:48:19.8840037Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8840680Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8841201Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8841618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9057
2022-12-01T10:48:19.8842186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9058
2022-12-01T10:48:19.8842941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8843279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8844041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8844412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8845171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8845516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8846280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8846663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8847128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8847616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8848438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8849235Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8849691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8850140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8851207Z STAGE:2022-12-01 10:24:16 9058:9058 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:24:16 9057:9057 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8851407Z 
2022-12-01T10:48:19.8852513Z STAGE:2022-12-01 10:24:16 9058:9058 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:24:16 9057:9057 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8852543Z 
2022-12-01T10:48:19.8853200Z STAGE:2022-12-01 10:24:16 9057:9057 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8853856Z STAGE:2022-12-01 10:24:16 9058:9058 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8854499Z STAGE:2022-12-01 10:24:16 9057:9057 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8855168Z STAGE:2022-12-01 10:24:16 9058:9058 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8855373Z ok (3.921s)
2022-12-01T10:48:19.8855401Z 
2022-12-01T10:48:19.8855928Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8856164Z Ran 1 test in 3.921s
2022-12-01T10:48:19.8856194Z 
2022-12-01T10:48:19.8856371Z OK
2022-12-01T10:48:19.8856398Z 
2022-12-01T10:48:19.8856648Z Generating XML reports...
2022-12-01T10:48:19.8857567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102413.xml
2022-12-01T10:48:19.8858329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8858656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8859433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8859808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8859837Z 
2022-12-01T10:48:19.8860043Z Running tests...
2022-12-01T10:48:19.8860555Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8861307Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8861815Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:48:19.8861846Z 
2022-12-01T10:48:19.8862358Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8862558Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8862608Z 
2022-12-01T10:48:19.8862787Z OK (skipped=1)
2022-12-01T10:48:19.8862815Z 
2022-12-01T10:48:19.8863043Z Generating XML reports...
2022-12-01T10:48:19.8863954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102419.xml
2022-12-01T10:48:19.8864715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8865055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8865838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8866201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8866231Z 
2022-12-01T10:48:19.8866438Z Running tests...
2022-12-01T10:48:19.8866954Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8867605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8868139Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T10:48:19.8868171Z 
2022-12-01T10:48:19.8868694Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8868906Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8868933Z 
2022-12-01T10:48:19.8869131Z OK (skipped=1)
2022-12-01T10:48:19.8869158Z 
2022-12-01T10:48:19.8869696Z Generating XML reports...
2022-12-01T10:48:19.8870826Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102421.xml
2022-12-01T10:48:19.8871609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8871938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8872713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8873089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8873123Z 
2022-12-01T10:48:19.8873332Z Running tests...
2022-12-01T10:48:19.8873861Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8874495Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8875010Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8875466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9521
2022-12-01T10:48:19.8875874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9522
2022-12-01T10:48:19.8876634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8876976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8877743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8878129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8878954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8879308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8880288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8880654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8881108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8881594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8882409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8883209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8883683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8884135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8884629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.8885097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.8885897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8886688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8887373Z STAGE:2022-12-01 10:24:27 9522:9522 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8888056Z STAGE:2022-12-01 10:24:27 9521:9521 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8888719Z STAGE:2022-12-01 10:24:27 9522:9522 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8889377Z STAGE:2022-12-01 10:24:27 9521:9521 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8890168Z STAGE:2022-12-01 10:24:27 9522:9522 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8890843Z STAGE:2022-12-01 10:24:27 9521:9521 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.8891519Z STAGE:2022-12-01 10:24:27 9521:9521 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8892192Z STAGE:2022-12-01 10:24:27 9522:9522 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.8892367Z ok (4.050s)
2022-12-01T10:48:19.8892398Z 
2022-12-01T10:48:19.8892916Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8893133Z Ran 1 test in 4.050s
2022-12-01T10:48:19.8893161Z 
2022-12-01T10:48:19.8893332Z OK
2022-12-01T10:48:19.8893359Z 
2022-12-01T10:48:19.8893597Z Generating XML reports...
2022-12-01T10:48:19.8894520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102423.xml
2022-12-01T10:48:19.8895310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8895665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8896411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8896798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8896829Z 
2022-12-01T10:48:19.8897035Z Running tests...
2022-12-01T10:48:19.8897548Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8898183Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8898691Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8899127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9871
2022-12-01T10:48:19.8899723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9872
2022-12-01T10:48:19.8900460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8900812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8901572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8901941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8902692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8903027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8903797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8904190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8904667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8905125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8905962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8906780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8907236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8907693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8907983Z skip: Skipped due to small world size. (3.948s)
2022-12-01T10:48:19.8908024Z 
2022-12-01T10:48:19.8908700Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8908939Z Ran 1 test in 3.948s
2022-12-01T10:48:19.8908969Z 
2022-12-01T10:48:19.8909172Z OK (skipped=1)
2022-12-01T10:48:19.8909198Z 
2022-12-01T10:48:19.8909699Z Generating XML reports...
2022-12-01T10:48:19.8910636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102429.xml
2022-12-01T10:48:19.8911415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8911755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8912529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8912907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8912938Z 
2022-12-01T10:48:19.8913156Z Running tests...
2022-12-01T10:48:19.8913677Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8914285Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8914887Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:48:19.8914925Z 
2022-12-01T10:48:19.8915448Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8915676Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8915704Z 
2022-12-01T10:48:19.8915904Z OK (skipped=1)
2022-12-01T10:48:19.8915932Z 
2022-12-01T10:48:19.8916168Z Generating XML reports...
2022-12-01T10:48:19.8917076Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102436.xml
2022-12-01T10:48:19.8917833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8918391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8919147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8919525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8919553Z 
2022-12-01T10:48:19.8919770Z Running tests...
2022-12-01T10:48:19.8920301Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8920946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8921530Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T10:48:19.8921561Z 
2022-12-01T10:48:19.8922085Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8922315Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8922354Z 
2022-12-01T10:48:19.8922559Z OK (skipped=1)
2022-12-01T10:48:19.8922586Z 
2022-12-01T10:48:19.8922803Z Generating XML reports...
2022-12-01T10:48:19.8923723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102438.xml
2022-12-01T10:48:19.8924488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8924831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8925614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8925986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8926016Z 
2022-12-01T10:48:19.8926220Z Running tests...
2022-12-01T10:48:19.8926760Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8927414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8928133Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:48:19.8928170Z 
2022-12-01T10:48:19.8928712Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8928935Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8928961Z 
2022-12-01T10:48:19.8929161Z OK (skipped=1)
2022-12-01T10:48:19.8929188Z 
2022-12-01T10:48:19.8929421Z Generating XML reports...
2022-12-01T10:48:19.8930335Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102440.xml
2022-12-01T10:48:19.8931094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8931444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8932221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8932589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8932619Z 
2022-12-01T10:48:19.8932825Z Running tests...
2022-12-01T10:48:19.8933355Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8933983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8934564Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T10:48:19.8934596Z 
2022-12-01T10:48:19.8935111Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8935322Z Ran 1 test in 0.002s
2022-12-01T10:48:19.8935351Z 
2022-12-01T10:48:19.8935533Z OK (skipped=1)
2022-12-01T10:48:19.8935559Z 
2022-12-01T10:48:19.8935788Z Generating XML reports...
2022-12-01T10:48:19.8936902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102442.xml
2022-12-01T10:48:19.8937673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8938023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8938787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8939157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8939196Z 
2022-12-01T10:48:19.8939397Z Running tests...
2022-12-01T10:48:19.8939906Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8940558Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8941078Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8941538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10482
2022-12-01T10:48:19.8941969Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10483
2022-12-01T10:48:19.8942739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8943078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8943841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8944211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8944979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8945316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8946060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8946578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8947075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8947557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8948343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8949160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8949918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8950389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8950576Z ok (3.992s)
2022-12-01T10:48:19.8950631Z 
2022-12-01T10:48:19.8951138Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8951366Z Ran 1 test in 3.993s
2022-12-01T10:48:19.8951397Z 
2022-12-01T10:48:19.8951585Z OK
2022-12-01T10:48:19.8951613Z 
2022-12-01T10:48:19.8951869Z Generating XML reports...
2022-12-01T10:48:19.8952836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102444.xml
2022-12-01T10:48:19.8953521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8953844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8954553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8954871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8955077Z 
2022-12-01T10:48:19.8955303Z Running tests...
2022-12-01T10:48:19.8955841Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8956478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8957022Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8957467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10762
2022-12-01T10:48:19.8957914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10763
2022-12-01T10:48:19.8958670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8958995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8959772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8960161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8960922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8961287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8962203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8962576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8963050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8963523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8964311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8965235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8965710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8966164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8966635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.8967105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.8967916Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8968731Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8969217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:19.8969691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:19.8970508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:19.8971311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:19.8971816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-12-01T10:48:19.8972305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-12-01T10:48:19.8973120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T10:48:19.8973936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T10:48:19.8974267Z ok (4.217s)
2022-12-01T10:48:19.8974302Z 
2022-12-01T10:48:19.8974839Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8975043Z Ran 1 test in 4.217s
2022-12-01T10:48:19.8975098Z 
2022-12-01T10:48:19.8975251Z OK
2022-12-01T10:48:19.8975279Z 
2022-12-01T10:48:19.8975521Z Generating XML reports...
2022-12-01T10:48:19.8976430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102450.xml
2022-12-01T10:48:19.8977213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8977565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8978417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8978797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8978838Z 
2022-12-01T10:48:19.8979050Z Running tests...
2022-12-01T10:48:19.8979552Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8980187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8980700Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-12-01T10:48:19.8980729Z 
2022-12-01T10:48:19.8981239Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8981453Z Ran 1 test in 0.003s
2022-12-01T10:48:19.8981481Z 
2022-12-01T10:48:19.8981681Z OK (skipped=1)
2022-12-01T10:48:19.8981709Z 
2022-12-01T10:48:19.8981936Z Generating XML reports...
2022-12-01T10:48:19.8982845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102457.xml
2022-12-01T10:48:19.8983604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8984049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8984847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8985228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8985259Z 
2022-12-01T10:48:19.8985476Z Running tests...
2022-12-01T10:48:19.8986006Z ----------------------------------------------------------------------
2022-12-01T10:48:19.8986642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.8987208Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.8987654Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11168
2022-12-01T10:48:19.8988063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11169
2022-12-01T10:48:19.8988862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8989211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8990457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8990841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8991592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.8991945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.8992696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.8993077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.8993725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.8994211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.8995046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8995849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.8996317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.8996770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.8997237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.8997710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.8998558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.8999370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9000040Z STAGE:2022-12-01 10:25:02 11168:11168 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9000703Z STAGE:2022-12-01 10:25:02 11169:11169 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9002251Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9002468Z   warnings.warn(
2022-12-01T10:48:19.9004148Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9004405Z   warnings.warn(
2022-12-01T10:48:19.9005522Z STAGE:2022-12-01 10:25:02 11168:11168 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:02 11169:11169 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9005558Z 
2022-12-01T10:48:19.9006207Z STAGE:2022-12-01 10:25:02 11168:11168 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9006851Z STAGE:2022-12-01 10:25:02 11169:11169 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9007916Z STAGE:2022-12-01 10:25:02 11169:11169 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:02 11168:11168 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9007960Z 
2022-12-01T10:48:19.9008155Z ok (4.016s)
2022-12-01T10:48:19.9008184Z 
2022-12-01T10:48:19.9008697Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9008918Z Ran 1 test in 4.016s
2022-12-01T10:48:19.9008946Z 
2022-12-01T10:48:19.9009117Z OK
2022-12-01T10:48:19.9009143Z 
2022-12-01T10:48:19.9009375Z Generating XML reports...
2022-12-01T10:48:19.9010288Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102459.xml
2022-12-01T10:48:19.9011057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9011390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9012139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9012516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9012677Z 
2022-12-01T10:48:19.9012906Z Running tests...
2022-12-01T10:48:19.9013446Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9014098Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9014665Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9015100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11432
2022-12-01T10:48:19.9015494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11433
2022-12-01T10:48:19.9016261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9016605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9017386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9017775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9018522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9018870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9019653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9020012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9020466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9020950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9021765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9022710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9023186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9023641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9024124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9024593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9025431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9026209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9026901Z STAGE:2022-12-01 10:25:08 11432:11432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9028389Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9028613Z   warnings.warn(
2022-12-01T10:48:19.9029284Z STAGE:2022-12-01 10:25:08 11433:11433 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9031085Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9031227Z   warnings.warn(
2022-12-01T10:48:19.9031592Z STAGE:2022-12-01 10:25:08 11432:11432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9032098Z STAGE:2022-12-01 10:25:08 11433:11433 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9032433Z STAGE:2022-12-01 10:25:08 11432:11432 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9032737Z STAGE:2022-12-01 10:25:08 11433:11433 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9033073Z STAGE:2022-12-01 10:25:08 11432:11432 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9033401Z STAGE:2022-12-01 10:25:08 11433:11433 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9033505Z ok (3.960s)
2022-12-01T10:48:19.9033528Z 
2022-12-01T10:48:19.9033798Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9033913Z Ran 1 test in 3.960s
2022-12-01T10:48:19.9033933Z 
2022-12-01T10:48:19.9034028Z OK
2022-12-01T10:48:19.9034047Z 
2022-12-01T10:48:19.9034177Z Generating XML reports...
2022-12-01T10:48:19.9034623Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102505.xml
2022-12-01T10:48:19.9034998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9035174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9035558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9035751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9035770Z 
2022-12-01T10:48:19.9035878Z Running tests...
2022-12-01T10:48:19.9036144Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9036460Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9036754Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9037041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11731
2022-12-01T10:48:19.9037268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11732
2022-12-01T10:48:19.9037644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9037819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9038198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9038387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9038752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9038925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9039290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9039482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9039730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9039977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9040376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9040772Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9041004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9041233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9041557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9041781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9042184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9042581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9042916Z STAGE:2022-12-01 10:25:15 11731:11731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9043652Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9043771Z   warnings.warn(
2022-12-01T10:48:19.9044104Z STAGE:2022-12-01 10:25:15 11732:11732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9044837Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9044949Z   warnings.warn(
2022-12-01T10:48:19.9045292Z STAGE:2022-12-01 10:25:15 11732:11732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9045602Z STAGE:2022-12-01 10:25:15 11731:11731 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9045928Z STAGE:2022-12-01 10:25:15 11732:11732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9046246Z STAGE:2022-12-01 10:25:15 11731:11731 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9046634Z STAGE:2022-12-01 10:25:15 11732:11732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9046971Z STAGE:2022-12-01 10:25:15 11731:11731 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9047073Z ok (4.168s)
2022-12-01T10:48:19.9047093Z 
2022-12-01T10:48:19.9047359Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9047472Z Ran 1 test in 4.169s
2022-12-01T10:48:19.9047492Z 
2022-12-01T10:48:19.9047585Z OK
2022-12-01T10:48:19.9047605Z 
2022-12-01T10:48:19.9047711Z Generating XML reports...
2022-12-01T10:48:19.9048165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102511.xml
2022-12-01T10:48:19.9048540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9048723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9049105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9049300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9049319Z 
2022-12-01T10:48:19.9049430Z Running tests...
2022-12-01T10:48:19.9049694Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9049992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9050278Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9050497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12072
2022-12-01T10:48:19.9050714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12073
2022-12-01T10:48:19.9051156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9051333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9051714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9051903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9052270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9052428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9052805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9052992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9053242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9053493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9053896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9054292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9054524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9054737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9054976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9055216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9055616Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9056071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9056419Z STAGE:2022-12-01 10:25:21 12072:12072 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9057159Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9057274Z   warnings.warn(
2022-12-01T10:48:19.9057608Z STAGE:2022-12-01 10:25:21 12073:12073 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9058340Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9058454Z   warnings.warn(
2022-12-01T10:48:19.9058775Z STAGE:2022-12-01 10:25:21 12073:12073 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9059105Z STAGE:2022-12-01 10:25:21 12072:12072 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9059637Z STAGE:2022-12-01 10:25:21 12072:12072 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:25:21 12073:12073 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9059658Z 
2022-12-01T10:48:19.9060195Z STAGE:2022-12-01 10:25:21 12072:12072 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:21 12073:12073 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9060216Z 
2022-12-01T10:48:19.9060376Z ok (4.022s)
2022-12-01T10:48:19.9060396Z 
2022-12-01T10:48:19.9060667Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9060783Z Ran 1 test in 4.022s
2022-12-01T10:48:19.9060803Z 
2022-12-01T10:48:19.9060897Z OK
2022-12-01T10:48:19.9060916Z 
2022-12-01T10:48:19.9061041Z Generating XML reports...
2022-12-01T10:48:19.9061476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102517.xml
2022-12-01T10:48:19.9061851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9062031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9062415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9062612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9062635Z 
2022-12-01T10:48:19.9062745Z Running tests...
2022-12-01T10:48:19.9063011Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9063325Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9063605Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9063808Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12286
2022-12-01T10:48:19.9064026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12287
2022-12-01T10:48:19.9064397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9064577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9064956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9065150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9065565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9065748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9066125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9066298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9066544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9066788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9067186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9067583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9067820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9068050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9068209Z skip: Skipped due to small world size. (3.760s)
2022-12-01T10:48:19.9068229Z 
2022-12-01T10:48:19.9068497Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9068591Z Ran 1 test in 3.761s
2022-12-01T10:48:19.9068611Z 
2022-12-01T10:48:19.9068715Z OK (skipped=1)
2022-12-01T10:48:19.9068734Z 
2022-12-01T10:48:19.9068858Z Generating XML reports...
2022-12-01T10:48:19.9069316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102524.xml
2022-12-01T10:48:19.9070121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9070413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9070805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9070997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9071018Z 
2022-12-01T10:48:19.9071108Z Running tests...
2022-12-01T10:48:19.9071373Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9071686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9071959Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9072180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12577
2022-12-01T10:48:19.9072395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12578
2022-12-01T10:48:19.9072776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9072954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9073335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9073509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9073874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9074044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9074421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9074611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9074861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9075175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9075591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9075967Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9076198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9076428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9076588Z skip: Skipped due to small world size. (3.855s)
2022-12-01T10:48:19.9076608Z 
2022-12-01T10:48:19.9076873Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9076989Z Ran 1 test in 3.856s
2022-12-01T10:48:19.9077009Z 
2022-12-01T10:48:19.9077117Z OK (skipped=1)
2022-12-01T10:48:19.9077139Z 
2022-12-01T10:48:19.9077265Z Generating XML reports...
2022-12-01T10:48:19.9077718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102530.xml
2022-12-01T10:48:19.9078071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9078251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9078687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9078879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9078899Z 
2022-12-01T10:48:19.9079007Z Running tests...
2022-12-01T10:48:19.9079273Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9079653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9079943Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9080163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12884
2022-12-01T10:48:19.9080364Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12885
2022-12-01T10:48:19.9080737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9080912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9081292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9081481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9081845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9082028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9082404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9082574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9082819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9083061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9083460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9083854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9084086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9084367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9084536Z skip: Skipped due to small world size. (4.141s)
2022-12-01T10:48:19.9084555Z 
2022-12-01T10:48:19.9084822Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9084917Z Ran 1 test in 4.141s
2022-12-01T10:48:19.9084936Z 
2022-12-01T10:48:19.9085045Z OK (skipped=1)
2022-12-01T10:48:19.9085064Z 
2022-12-01T10:48:19.9085187Z Generating XML reports...
2022-12-01T10:48:19.9085640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102536.xml
2022-12-01T10:48:19.9086006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9086185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9086574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9086767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9086787Z 
2022-12-01T10:48:19.9086893Z Running tests...
2022-12-01T10:48:19.9087140Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9087452Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9087732Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9087952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13142
2022-12-01T10:48:19.9088168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13143
2022-12-01T10:48:19.9088541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9088847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9089236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9089408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9089774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9089950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9090327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9090511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9090758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9091000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9091404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9091803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9092016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9092245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9092404Z skip: Skipped due to small world size. (3.922s)
2022-12-01T10:48:19.9092424Z 
2022-12-01T10:48:19.9092688Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9092803Z Ran 1 test in 3.922s
2022-12-01T10:48:19.9092822Z 
2022-12-01T10:48:19.9092930Z OK (skipped=1)
2022-12-01T10:48:19.9092949Z 
2022-12-01T10:48:19.9093074Z Generating XML reports...
2022-12-01T10:48:19.9093581Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102543.xml
2022-12-01T10:48:19.9093948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9094125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9094506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9094700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9094720Z 
2022-12-01T10:48:19.9094830Z Running tests...
2022-12-01T10:48:19.9095100Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9095414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9095685Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9095911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13453
2022-12-01T10:48:19.9096110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13454
2022-12-01T10:48:19.9096482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9096657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9097028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9097205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9097585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9097777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9098215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9098388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9098631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9098874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9099274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9099667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9099896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9100124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9100468Z STAGE:2022-12-01 10:25:52 13454:13454 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9100794Z STAGE:2022-12-01 10:25:52 13453:13453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9101538Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9101633Z   warnings.warn(
2022-12-01T10:48:19.9102367Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9102481Z   warnings.warn(
2022-12-01T10:48:19.9103086Z STAGE:2022-12-01 10:25:52 13454:13454 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:25:52 13453:13453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9103108Z 
2022-12-01T10:48:19.9103643Z STAGE:2022-12-01 10:25:52 13454:13454 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:25:52 13453:13453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9103663Z 
2022-12-01T10:48:19.9104000Z STAGE:2022-12-01 10:25:52 13453:13453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9104330Z STAGE:2022-12-01 10:25:52 13454:13454 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9104434Z ok (3.941s)
2022-12-01T10:48:19.9104453Z 
2022-12-01T10:48:19.9104718Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9104833Z Ran 1 test in 3.942s
2022-12-01T10:48:19.9104853Z 
2022-12-01T10:48:19.9104932Z OK
2022-12-01T10:48:19.9104969Z 
2022-12-01T10:48:19.9105076Z Generating XML reports...
2022-12-01T10:48:19.9105534Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102549.xml
2022-12-01T10:48:19.9105906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9106084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9106464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9106657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9106676Z 
2022-12-01T10:48:19.9106786Z Running tests...
2022-12-01T10:48:19.9107050Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9107346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9107719Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9107938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13768
2022-12-01T10:48:19.9108154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13769
2022-12-01T10:48:19.9108530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9108706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9109085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9109275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9110016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9110187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9110576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9110767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9111012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9111254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9111653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9112048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9112279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9113113Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9113222Z   warnings.warn(
2022-12-01T10:48:19.9113455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9114193Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9114304Z   warnings.warn(
2022-12-01T10:48:19.9114406Z ok (4.120s)
2022-12-01T10:48:19.9114427Z 
2022-12-01T10:48:19.9114692Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9114808Z Ran 1 test in 4.120s
2022-12-01T10:48:19.9114828Z 
2022-12-01T10:48:19.9114919Z OK
2022-12-01T10:48:19.9114941Z 
2022-12-01T10:48:19.9115065Z Generating XML reports...
2022-12-01T10:48:19.9115502Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102555.xml
2022-12-01T10:48:19.9115874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9116050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9116427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9116619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9116639Z 
2022-12-01T10:48:19.9116744Z Running tests...
2022-12-01T10:48:19.9117006Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9117396Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9117647Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9117866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14079
2022-12-01T10:48:19.9118083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14080
2022-12-01T10:48:19.9118454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9118626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9119003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9119190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9119554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9119734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9120090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9120277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9120522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9120765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9121162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9121559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9121793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9122210Z STAGE:2022-12-01 10:26:05 14080:14080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9122966Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9123063Z   warnings.warn(
2022-12-01T10:48:19.9123292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9123625Z STAGE:2022-12-01 10:26:05 14079:14079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9124361Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9124477Z   warnings.warn(
2022-12-01T10:48:19.9124816Z STAGE:2022-12-01 10:26:05 14079:14079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9125145Z STAGE:2022-12-01 10:26:05 14080:14080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9125469Z STAGE:2022-12-01 10:26:05 14079:14079 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9125789Z STAGE:2022-12-01 10:26:05 14080:14080 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9126104Z STAGE:2022-12-01 10:26:05 14079:14079 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9126431Z STAGE:2022-12-01 10:26:05 14080:14080 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9126535Z ok (3.992s)
2022-12-01T10:48:19.9126555Z 
2022-12-01T10:48:19.9126881Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9126998Z Ran 1 test in 3.993s
2022-12-01T10:48:19.9127017Z 
2022-12-01T10:48:19.9127109Z OK
2022-12-01T10:48:19.9127128Z 
2022-12-01T10:48:19.9127253Z Generating XML reports...
2022-12-01T10:48:19.9127711Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102601.xml
2022-12-01T10:48:19.9128087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9128249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9128629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9128822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9128842Z 
2022-12-01T10:48:19.9128951Z Running tests...
2022-12-01T10:48:19.9129218Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9129534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9129813Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9130034Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14394
2022-12-01T10:48:19.9130235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14395
2022-12-01T10:48:19.9130608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9130782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9131161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9131351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9131763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9131943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9132318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9132505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9132733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9132977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9133374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9133769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9134006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9134233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9134569Z STAGE:2022-12-01 10:26:11 14395:14395 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9134892Z STAGE:2022-12-01 10:26:11 14394:14394 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9135627Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9135723Z   warnings.warn(
2022-12-01T10:48:19.9136457Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9136623Z   warnings.warn(
2022-12-01T10:48:19.9137175Z STAGE:2022-12-01 10:26:11 14394:14394 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:11 14395:14395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9137195Z 
2022-12-01T10:48:19.9137727Z STAGE:2022-12-01 10:26:11 14395:14395 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:26:11 14394:14394 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9137748Z 
2022-12-01T10:48:19.9138081Z STAGE:2022-12-01 10:26:11 14394:14394 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9138409Z STAGE:2022-12-01 10:26:11 14395:14395 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9138515Z ok (3.824s)
2022-12-01T10:48:19.9138535Z 
2022-12-01T10:48:19.9138804Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9138917Z Ran 1 test in 3.824s
2022-12-01T10:48:19.9138937Z 
2022-12-01T10:48:19.9139030Z OK
2022-12-01T10:48:19.9139049Z 
2022-12-01T10:48:19.9139155Z Generating XML reports...
2022-12-01T10:48:19.9139607Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102607.xml
2022-12-01T10:48:19.9139982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9140161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9140540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9140731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9140754Z 
2022-12-01T10:48:19.9140863Z Running tests...
2022-12-01T10:48:19.9141169Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9141474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9141746Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9141967Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14689
2022-12-01T10:48:19.9142186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14690
2022-12-01T10:48:19.9142558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9142734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9143113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9143314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9143683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9143839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9144209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9144395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9144642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9144887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9145289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9145746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9145974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9146311Z STAGE:2022-12-01 10:26:17 14690:14690 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9147032Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9147147Z   warnings.warn(
2022-12-01T10:48:19.9147374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9147706Z STAGE:2022-12-01 10:26:17 14689:14689 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9148443Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T10:48:19.9148554Z   warnings.warn(
2022-12-01T10:48:19.9148893Z STAGE:2022-12-01 10:26:17 14689:14689 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9149222Z STAGE:2022-12-01 10:26:17 14690:14690 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9149968Z STAGE:2022-12-01 10:26:17 14689:14689 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9150297Z STAGE:2022-12-01 10:26:17 14690:14690 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9150839Z STAGE:2022-12-01 10:26:17 14689:14689 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:17 14690:14690 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9150866Z 
2022-12-01T10:48:19.9151054Z ok (3.990s)
2022-12-01T10:48:19.9151074Z 
2022-12-01T10:48:19.9151355Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9151466Z Ran 1 test in 3.990s
2022-12-01T10:48:19.9151485Z 
2022-12-01T10:48:19.9151578Z OK
2022-12-01T10:48:19.9151599Z 
2022-12-01T10:48:19.9151723Z Generating XML reports...
2022-12-01T10:48:19.9152174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102613.xml
2022-12-01T10:48:19.9152543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9152721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9153083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9153280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9153303Z 
2022-12-01T10:48:19.9153414Z Running tests...
2022-12-01T10:48:19.9153680Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9153996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9154282Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9154503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15000
2022-12-01T10:48:19.9154720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15001
2022-12-01T10:48:19.9155073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9155250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9155635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9155898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9156268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9156443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9156816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9157006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9157252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9157479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9157879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9158285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9158516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9158743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9158842Z ok (3.844s)
2022-12-01T10:48:19.9158862Z 
2022-12-01T10:48:19.9159129Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9159244Z Ran 1 test in 3.844s
2022-12-01T10:48:19.9159264Z 
2022-12-01T10:48:19.9159339Z OK
2022-12-01T10:48:19.9159373Z 
2022-12-01T10:48:19.9159479Z Generating XML reports...
2022-12-01T10:48:19.9159930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102620.xml
2022-12-01T10:48:19.9160350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9160536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9160914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9161106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9161125Z 
2022-12-01T10:48:19.9161233Z Running tests...
2022-12-01T10:48:19.9161492Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9161788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9162062Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9162282Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15236
2022-12-01T10:48:19.9162504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15237
2022-12-01T10:48:19.9162878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9163053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9163433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9163622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9163966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9164140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9164511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9164698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9165003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9165249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9165650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9166044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9166277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9166488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9166727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9166968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9167373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9167764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9168099Z STAGE:2022-12-01 10:26:29 15236:15236 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9168422Z STAGE:2022-12-01 10:26:29 15237:15237 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9168759Z STAGE:2022-12-01 10:26:29 15236:15236 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9169086Z STAGE:2022-12-01 10:26:29 15237:15237 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9169396Z STAGE:2022-12-01 10:26:29 15236:15236 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9169770Z STAGE:2022-12-01 10:26:29 15237:15237 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9170117Z STAGE:2022-12-01 10:26:29 15237:15237 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9170445Z STAGE:2022-12-01 10:26:29 15236:15236 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9170546Z ok (4.018s)
2022-12-01T10:48:19.9170565Z 
2022-12-01T10:48:19.9170830Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9170943Z Ran 1 test in 4.018s
2022-12-01T10:48:19.9170963Z 
2022-12-01T10:48:19.9171056Z OK
2022-12-01T10:48:19.9171075Z 
2022-12-01T10:48:19.9171200Z Generating XML reports...
2022-12-01T10:48:19.9171633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102626.xml
2022-12-01T10:48:19.9172005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9172186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9172565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9172757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9172776Z 
2022-12-01T10:48:19.9172884Z Running tests...
2022-12-01T10:48:19.9173148Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9173456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9173707Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9173927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15525
2022-12-01T10:48:19.9174139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15526
2022-12-01T10:48:19.9174586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9174761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9175142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9175334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9175694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9175866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9176220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9176409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9176658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9176902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9177305Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9177700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9177930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9178157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9178448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9178675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9179127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9179528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9179866Z STAGE:2022-12-01 10:26:36 15526:15526 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9180191Z STAGE:2022-12-01 10:26:36 15525:15525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9180736Z STAGE:2022-12-01 10:26:36 15525:15525 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:36 15526:15526 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9180757Z 
2022-12-01T10:48:19.9181085Z STAGE:2022-12-01 10:26:36 15526:15526 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9181410Z STAGE:2022-12-01 10:26:36 15525:15525 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9181757Z STAGE:2022-12-01 10:26:36 15526:15526 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9182085Z STAGE:2022-12-01 10:26:36 15525:15525 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9182170Z ok (4.071s)
2022-12-01T10:48:19.9182189Z 
2022-12-01T10:48:19.9182453Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9182566Z Ran 1 test in 4.071s
2022-12-01T10:48:19.9182585Z 
2022-12-01T10:48:19.9182678Z OK
2022-12-01T10:48:19.9182697Z 
2022-12-01T10:48:19.9182820Z Generating XML reports...
2022-12-01T10:48:19.9183272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102632.xml
2022-12-01T10:48:19.9183640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9183869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9184240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9184433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9184453Z 
2022-12-01T10:48:19.9184561Z Running tests...
2022-12-01T10:48:19.9184823Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9185141Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9185421Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9185644Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15829
2022-12-01T10:48:19.9185856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15830
2022-12-01T10:48:19.9186228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9186393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9186772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9186962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9187326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9187500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9187871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9188059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9188306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9188581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9188995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9189722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9190038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9190266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9190507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9190748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9191158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9191560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9191879Z STAGE:2022-12-01 10:26:42 15829:15829 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9192205Z STAGE:2022-12-01 10:26:42 15830:15830 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9192538Z STAGE:2022-12-01 10:26:42 15829:15829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9192863Z STAGE:2022-12-01 10:26:42 15830:15830 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9193186Z STAGE:2022-12-01 10:26:42 15830:15830 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9193518Z STAGE:2022-12-01 10:26:42 15829:15829 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9193956Z STAGE:2022-12-01 10:26:42 15829:15829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9194282Z STAGE:2022-12-01 10:26:42 15830:15830 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9194387Z ok (4.184s)
2022-12-01T10:48:19.9194407Z 
2022-12-01T10:48:19.9194652Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9194764Z Ran 1 test in 4.184s
2022-12-01T10:48:19.9194784Z 
2022-12-01T10:48:19.9194876Z OK
2022-12-01T10:48:19.9194895Z 
2022-12-01T10:48:19.9195019Z Generating XML reports...
2022-12-01T10:48:19.9195472Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102638.xml
2022-12-01T10:48:19.9195845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9196024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9196414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9196607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9196627Z 
2022-12-01T10:48:19.9196718Z Running tests...
2022-12-01T10:48:19.9196981Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9197292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9197561Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9197782Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16132
2022-12-01T10:48:19.9198002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16133
2022-12-01T10:48:19.9198373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9198616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9198995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9199185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9199548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9199721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9200096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9200285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9200532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9200781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9201185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9201561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9201792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9202018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9202258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9202500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9202896Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9203374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9203714Z STAGE:2022-12-01 10:26:49 16133:16133 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9204040Z STAGE:2022-12-01 10:26:49 16132:16132 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9204569Z STAGE:2022-12-01 10:26:49 16133:16133 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:49 16132:16132 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9204609Z 
2022-12-01T10:48:19.9205121Z STAGE:2022-12-01 10:26:49 16133:16133 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:26:49 16132:16132 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9205159Z 
2022-12-01T10:48:19.9205685Z STAGE:2022-12-01 10:26:49 16132:16132 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:26:49 16133:16133 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9205709Z 
2022-12-01T10:48:19.9205812Z ok (4.204s)
2022-12-01T10:48:19.9205832Z 
2022-12-01T10:48:19.9206095Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9206211Z Ran 1 test in 4.204s
2022-12-01T10:48:19.9206231Z 
2022-12-01T10:48:19.9206324Z OK
2022-12-01T10:48:19.9206343Z 
2022-12-01T10:48:19.9206468Z Generating XML reports...
2022-12-01T10:48:19.9206916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102645.xml
2022-12-01T10:48:19.9207289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9207465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9207827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9208072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9208093Z 
2022-12-01T10:48:19.9208206Z Running tests...
2022-12-01T10:48:19.9208469Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9208782Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9209043Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9209260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16416
2022-12-01T10:48:19.9209479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16417
2022-12-01T10:48:19.9209832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9210008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9210396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9210588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9210950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9211124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9211496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9211684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9211931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9212158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9212618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9213011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9213241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9213469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9213630Z skip: Skipped due to small world size. (3.867s)
2022-12-01T10:48:19.9213649Z 
2022-12-01T10:48:19.9213914Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9214027Z Ran 1 test in 3.867s
2022-12-01T10:48:19.9214046Z 
2022-12-01T10:48:19.9214135Z OK (skipped=1)
2022-12-01T10:48:19.9214171Z 
2022-12-01T10:48:19.9214277Z Generating XML reports...
2022-12-01T10:48:19.9214728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102651.xml
2022-12-01T10:48:19.9215102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9215280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9215659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9215852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9215871Z 
2022-12-01T10:48:19.9215982Z Running tests...
2022-12-01T10:48:19.9216245Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9216541Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9216805Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9217075Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16693
2022-12-01T10:48:19.9217299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16694
2022-12-01T10:48:19.9217669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9217844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9218222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9218413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9218757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9218932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9219305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9219498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9219746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9219988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9220385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9220779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9221012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9221222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9221383Z skip: Skipped due to small world size. (4.018s)
2022-12-01T10:48:19.9221450Z 
2022-12-01T10:48:19.9221726Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9221839Z Ran 1 test in 4.018s
2022-12-01T10:48:19.9221858Z 
2022-12-01T10:48:19.9221965Z OK (skipped=1)
2022-12-01T10:48:19.9221984Z 
2022-12-01T10:48:19.9222108Z Generating XML reports...
2022-12-01T10:48:19.9222564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102657.xml
2022-12-01T10:48:19.9222936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9223109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9223469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9223661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9223684Z 
2022-12-01T10:48:19.9223794Z Running tests...
2022-12-01T10:48:19.9224059Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9224373Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9224644Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9224863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16999
2022-12-01T10:48:19.9225081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17000
2022-12-01T10:48:19.9225433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9225612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9225988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9226181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9226587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9226766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9227147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9227334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9227579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9227805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9228203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9228608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9228838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9229068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9229227Z skip: Skipped due to small world size. (3.887s)
2022-12-01T10:48:19.9229246Z 
2022-12-01T10:48:19.9229876Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9230002Z Ran 1 test in 3.887s
2022-12-01T10:48:19.9230022Z 
2022-12-01T10:48:19.9230131Z OK (skipped=1)
2022-12-01T10:48:19.9230151Z 
2022-12-01T10:48:19.9230257Z Generating XML reports...
2022-12-01T10:48:19.9230723Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102704.xml
2022-12-01T10:48:19.9231094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9231482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9231911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9232133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9232153Z 
2022-12-01T10:48:19.9232299Z Running tests...
2022-12-01T10:48:19.9232602Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9232902Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9233204Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9233457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17236
2022-12-01T10:48:19.9233688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17237
2022-12-01T10:48:19.9234251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9234469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9234891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9235118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9235518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9235728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9236088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9236309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9236776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9237117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9237565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9237996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9238261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9238528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9238670Z skip: Skipped due to small world size. (3.974s)
2022-12-01T10:48:19.9238754Z 
2022-12-01T10:48:19.9239004Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9239160Z Ran 1 test in 3.974s
2022-12-01T10:48:19.9239181Z 
2022-12-01T10:48:19.9239322Z OK (skipped=1)
2022-12-01T10:48:19.9239345Z 
2022-12-01T10:48:19.9239540Z Generating XML reports...
2022-12-01T10:48:19.9240031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102710.xml
2022-12-01T10:48:19.9240437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9240648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9241071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9241248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9241318Z 
2022-12-01T10:48:19.9241411Z Running tests...
2022-12-01T10:48:19.9241707Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9242167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9242497Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9242754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17550
2022-12-01T10:48:19.9243005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17551
2022-12-01T10:48:19.9243429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9243644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9244010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9244234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9244633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9244850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9245294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9245520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9245808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9246086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9246470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9246901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9247167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9247513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9247899Z STAGE:2022-12-01 10:27:20 17551:17551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9248306Z STAGE:2022-12-01 10:27:20 17550:17550 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9248892Z STAGE:2022-12-01 10:27:20 17551:17551 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:27:20 17550:17550 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9248913Z 
2022-12-01T10:48:19.9249478Z STAGE:2022-12-01 10:27:20 17550:17550 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:27:20 17551:17551 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9249499Z 
2022-12-01T10:48:19.9249874Z STAGE:2022-12-01 10:27:20 17551:17551 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9250250Z STAGE:2022-12-01 10:27:20 17550:17550 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9250387Z ok (3.848s)
2022-12-01T10:48:19.9250407Z 
2022-12-01T10:48:19.9250657Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9250806Z Ran 1 test in 3.849s
2022-12-01T10:48:19.9250825Z 
2022-12-01T10:48:19.9250960Z OK
2022-12-01T10:48:19.9250979Z 
2022-12-01T10:48:19.9251174Z Generating XML reports...
2022-12-01T10:48:19.9251665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102716.xml
2022-12-01T10:48:19.9252071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9252281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9252698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9252966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9253038Z 
2022-12-01T10:48:19.9253132Z Running tests...
2022-12-01T10:48:19.9253446Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9253796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9254124Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9254381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17813
2022-12-01T10:48:19.9254634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17814
2022-12-01T10:48:19.9255038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9255251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9255622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9255856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9256257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9256465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9256907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9257131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9257413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9257689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9258174Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9258622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9258887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9259151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9259623Z STAGE:2022-12-01 10:27:26 17813:17813 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9260023Z STAGE:2022-12-01 10:27:26 17814:17814 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9260408Z STAGE:2022-12-01 10:27:26 17813:17813 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9260773Z STAGE:2022-12-01 10:27:26 17814:17814 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9261145Z STAGE:2022-12-01 10:27:26 17813:17813 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9261451Z STAGE:2022-12-01 10:27:26 17814:17814 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9261816Z STAGE:2022-12-01 10:27:26 17813:17813 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9262182Z STAGE:2022-12-01 10:27:26 17814:17814 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9262319Z ok (3.954s)
2022-12-01T10:48:19.9262339Z 
2022-12-01T10:48:19.9262637Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9262827Z Ran 1 test in 3.954s
2022-12-01T10:48:19.9262847Z 
2022-12-01T10:48:19.9262974Z OK
2022-12-01T10:48:19.9262994Z 
2022-12-01T10:48:19.9263185Z Generating XML reports...
2022-12-01T10:48:19.9263677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102722.xml
2022-12-01T10:48:19.9264094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9264308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9264724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9264952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9264972Z 
2022-12-01T10:48:19.9265125Z Running tests...
2022-12-01T10:48:19.9265483Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9265835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9266129Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9266335Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18129
2022-12-01T10:48:19.9266590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18130
2022-12-01T10:48:19.9266996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9267205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9267627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9267853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9268287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9268529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9268941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9269163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9269822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9270144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9270599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9271030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9271298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9271615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9271994Z STAGE:2022-12-01 10:27:34 18130:18130 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9272814Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:48:19.9272913Z   warnings.warn(
2022-12-01T10:48:19.9273296Z STAGE:2022-12-01 10:27:34 18129:18129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9274091Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:48:19.9274237Z   warnings.warn(
2022-12-01T10:48:19.9274611Z STAGE:2022-12-01 10:27:34 18129:18129 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9275122Z STAGE:2022-12-01 10:27:34 18130:18130 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9275528Z STAGE:2022-12-01 10:27:34 18129:18129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9275887Z STAGE:2022-12-01 10:27:34 18130:18130 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9276268Z STAGE:2022-12-01 10:27:34 18129:18129 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9276576Z STAGE:2022-12-01 10:27:34 18130:18130 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9276714Z ok (5.873s)
2022-12-01T10:48:19.9276736Z 
2022-12-01T10:48:19.9277034Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9277181Z Ran 1 test in 5.873s
2022-12-01T10:48:19.9277201Z 
2022-12-01T10:48:19.9277326Z OK
2022-12-01T10:48:19.9277345Z 
2022-12-01T10:48:19.9277506Z Generating XML reports...
2022-12-01T10:48:19.9278035Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102728.xml
2022-12-01T10:48:19.9278513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9278729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9279097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9279325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9279345Z 
2022-12-01T10:48:19.9279486Z Running tests...
2022-12-01T10:48:19.9279783Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9280132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9280504Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9280876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18457
2022-12-01T10:48:19.9281144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18458
2022-12-01T10:48:19.9281508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9281720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9282131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9282360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9282759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9282978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9283398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9283659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9283941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9284169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9284602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9285036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9285300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9285574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9286046Z STAGE:2022-12-01 10:27:42 18458:18458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9286850Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:48:19.9287033Z   warnings.warn(
2022-12-01T10:48:19.9287408Z STAGE:2022-12-01 10:27:42 18457:18457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9288208Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:48:19.9288304Z   warnings.warn(
2022-12-01T10:48:19.9288685Z STAGE:2022-12-01 10:27:42 18457:18457 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9289065Z STAGE:2022-12-01 10:27:42 18458:18458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9289428Z STAGE:2022-12-01 10:27:42 18457:18457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9289785Z STAGE:2022-12-01 10:27:42 18458:18458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9290150Z STAGE:2022-12-01 10:27:42 18457:18457 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9290541Z STAGE:2022-12-01 10:27:42 18458:18458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9290682Z ok (5.929s)
2022-12-01T10:48:19.9290703Z 
2022-12-01T10:48:19.9291003Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9291101Z Ran 1 test in 5.929s
2022-12-01T10:48:19.9291124Z 
2022-12-01T10:48:19.9291260Z OK
2022-12-01T10:48:19.9291279Z 
2022-12-01T10:48:19.9291518Z Generating XML reports...
2022-12-01T10:48:19.9292024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102736.xml
2022-12-01T10:48:19.9292432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9292645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9293099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9293339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9293359Z 
2022-12-01T10:48:19.9293451Z Running tests...
2022-12-01T10:48:19.9293752Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9294103Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9294405Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9294664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18811
2022-12-01T10:48:19.9314618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18812
2022-12-01T10:48:19.9315096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9315274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9315661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9315841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9316206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9316541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9316923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9317115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9317361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9317605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9318010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9318409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9318625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9318974Z STAGE:2022-12-01 10:27:48 18812:18812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9319200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9319525Z STAGE:2022-12-01 10:27:48 18811:18811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9319858Z STAGE:2022-12-01 10:27:48 18811:18811 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9320191Z STAGE:2022-12-01 10:27:48 18812:18812 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9320519Z STAGE:2022-12-01 10:27:48 18811:18811 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9320837Z STAGE:2022-12-01 10:27:48 18812:18812 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9321166Z STAGE:2022-12-01 10:27:48 18811:18811 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9321540Z STAGE:2022-12-01 10:27:48 18812:18812 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9321659Z ok (3.942s)
2022-12-01T10:48:19.9321682Z 
2022-12-01T10:48:19.9321956Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9322067Z Ran 1 test in 3.942s
2022-12-01T10:48:19.9322087Z 
2022-12-01T10:48:19.9322183Z OK
2022-12-01T10:48:19.9322202Z 
2022-12-01T10:48:19.9322320Z Generating XML reports...
2022-12-01T10:48:19.9322770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102745.xml
2022-12-01T10:48:19.9323145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9323322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9323690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9323891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9323912Z 
2022-12-01T10:48:19.9324019Z Running tests...
2022-12-01T10:48:19.9324282Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9324597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9324864Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9325085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19090
2022-12-01T10:48:19.9325304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19091
2022-12-01T10:48:19.9325661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9325839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9326285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9326478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9326831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9326997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9327370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9327557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9327804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9328029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9328436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9328839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9329068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9329289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9329387Z ok (5.653s)
2022-12-01T10:48:19.9329407Z 
2022-12-01T10:48:19.9329672Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9329777Z Ran 1 test in 5.654s
2022-12-01T10:48:19.9329796Z 
2022-12-01T10:48:19.9329871Z OK
2022-12-01T10:48:19.9329903Z 
2022-12-01T10:48:19.9330011Z Generating XML reports...
2022-12-01T10:48:19.9330465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102751.xml
2022-12-01T10:48:19.9330874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9331053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9331436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9331619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9331639Z 
2022-12-01T10:48:19.9331741Z Running tests...
2022-12-01T10:48:19.9332000Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9332296Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9332539Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9332752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19422
2022-12-01T10:48:19.9332975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19423
2022-12-01T10:48:19.9333340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9333512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9333891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9334072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9334420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9334589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9334963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9335206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9335446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9335684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9336090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9336477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9336699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9337020Z STAGE:2022-12-01 10:28:02 19423:19423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9337249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9337580Z STAGE:2022-12-01 10:28:02 19422:19422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9337910Z STAGE:2022-12-01 10:28:02 19422:19422 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9338237Z STAGE:2022-12-01 10:28:02 19423:19423 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9338551Z STAGE:2022-12-01 10:28:02 19422:19422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9338864Z STAGE:2022-12-01 10:28:02 19423:19423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9339410Z STAGE:2022-12-01 10:28:02 19423:19423 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:28:02 19422:19422 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9339431Z 
2022-12-01T10:48:19.9339532Z ok (4.128s)
2022-12-01T10:48:19.9339552Z 
2022-12-01T10:48:19.9339816Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9339916Z Ran 1 test in 4.128s
2022-12-01T10:48:19.9339992Z 
2022-12-01T10:48:19.9340091Z OK
2022-12-01T10:48:19.9340110Z 
2022-12-01T10:48:19.9340235Z Generating XML reports...
2022-12-01T10:48:19.9340682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102759.xml
2022-12-01T10:48:19.9341050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9341226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9341598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9341785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9341804Z 
2022-12-01T10:48:19.9341895Z Running tests...
2022-12-01T10:48:19.9342161Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9342487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9342742Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9342956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19732
2022-12-01T10:48:19.9343172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19733
2022-12-01T10:48:19.9343532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9343701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9344060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9344254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9344670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9344836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9345207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9345395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9345631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9345868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9346267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9346648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9346880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9347215Z STAGE:2022-12-01 10:28:08 19733:19733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9347444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9347769Z STAGE:2022-12-01 10:28:09 19732:19732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9348098Z STAGE:2022-12-01 10:28:09 19732:19732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9348430Z STAGE:2022-12-01 10:28:09 19733:19733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9348748Z STAGE:2022-12-01 10:28:09 19732:19732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9349061Z STAGE:2022-12-01 10:28:09 19733:19733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9349769Z STAGE:2022-12-01 10:28:09 19733:19733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9350175Z STAGE:2022-12-01 10:28:09 19732:19732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9350273Z ok (4.115s)
2022-12-01T10:48:19.9350294Z 
2022-12-01T10:48:19.9350557Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9350669Z Ran 1 test in 4.115s
2022-12-01T10:48:19.9350689Z 
2022-12-01T10:48:19.9350784Z OK
2022-12-01T10:48:19.9350803Z 
2022-12-01T10:48:19.9350919Z Generating XML reports...
2022-12-01T10:48:19.9351368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102805.xml
2022-12-01T10:48:19.9351741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9351899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9352280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9352467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9352487Z 
2022-12-01T10:48:19.9352585Z Running tests...
2022-12-01T10:48:19.9352841Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9353156Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9353412Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9353625Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20051
2022-12-01T10:48:19.9353824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20052
2022-12-01T10:48:19.9354196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9354444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9354825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9355018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9355375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9355542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9355916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9356096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9356325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9356564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9356967Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9357358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9357587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9357913Z STAGE:2022-12-01 10:28:15 20052:20052 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9358231Z STAGE:2022-12-01 10:28:15 20051:20051 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9358459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9358787Z STAGE:2022-12-01 10:28:15 20052:20052 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9359107Z STAGE:2022-12-01 10:28:15 20051:20051 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9359468Z STAGE:2022-12-01 10:28:15 20052:20052 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9359800Z STAGE:2022-12-01 10:28:15 20051:20051 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9360117Z STAGE:2022-12-01 10:28:15 20052:20052 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9360434Z STAGE:2022-12-01 10:28:15 20051:20051 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9360535Z ok (4.160s)
2022-12-01T10:48:19.9360554Z 
2022-12-01T10:48:19.9360811Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9360918Z Ran 1 test in 4.161s
2022-12-01T10:48:19.9360938Z 
2022-12-01T10:48:19.9361012Z OK
2022-12-01T10:48:19.9361030Z 
2022-12-01T10:48:19.9361156Z Generating XML reports...
2022-12-01T10:48:19.9361611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102811.xml
2022-12-01T10:48:19.9361976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9362151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9362523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9362707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9362726Z 
2022-12-01T10:48:19.9362835Z Running tests...
2022-12-01T10:48:19.9363089Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9363388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9363644Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9363918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20351
2022-12-01T10:48:19.9364126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20352
2022-12-01T10:48:19.9364492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9364661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9365031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9365225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9365571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9365733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9366097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9366294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9366530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9366770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9367166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9367556Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9367779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9367989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9368325Z STAGE:2022-12-01 10:28:23 20352:20352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9368689Z STAGE:2022-12-01 10:28:23 20351:20351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9369033Z STAGE:2022-12-01 10:28:23 20351:20351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9369356Z STAGE:2022-12-01 10:28:23 20352:20352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9369675Z STAGE:2022-12-01 10:28:23 20352:20352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9369996Z STAGE:2022-12-01 10:28:23 20351:20351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9370319Z STAGE:2022-12-01 10:28:24 20352:20352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9370641Z STAGE:2022-12-01 10:28:24 20351:20351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9370960Z STAGE:2022-12-01 10:28:24 20352:20352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9371279Z STAGE:2022-12-01 10:28:24 20351:20351 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9371613Z STAGE:2022-12-01 10:28:24 20352:20352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9371937Z STAGE:2022-12-01 10:28:24 20351:20351 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9372039Z ok (7.783s)
2022-12-01T10:48:19.9372058Z 
2022-12-01T10:48:19.9372322Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9372433Z Ran 1 test in 7.783s
2022-12-01T10:48:19.9372453Z 
2022-12-01T10:48:19.9372544Z OK
2022-12-01T10:48:19.9372563Z 
2022-12-01T10:48:19.9372668Z Generating XML reports...
2022-12-01T10:48:19.9373120Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102817.xml
2022-12-01T10:48:19.9373589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9373764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9374146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9374338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9374358Z 
2022-12-01T10:48:19.9374466Z Running tests...
2022-12-01T10:48:19.9374731Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9375046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9375299Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9375517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20720
2022-12-01T10:48:19.9375741Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20721
2022-12-01T10:48:19.9376113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9376288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9376666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9376858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9377217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9377374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9377750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9377943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9378230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9378536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9378947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9379344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9379572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9379798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9380112Z STAGE:2022-12-01 10:28:33 20720:20720 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9380483Z STAGE:2022-12-01 10:28:33 20721:20721 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9380817Z STAGE:2022-12-01 10:28:33 20720:20720 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9381145Z STAGE:2022-12-01 10:28:33 20721:20721 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9381467Z STAGE:2022-12-01 10:28:33 20720:20720 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9381796Z STAGE:2022-12-01 10:28:33 20721:20721 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9382339Z STAGE:2022-12-01 10:28:34 20721:20721 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:28:34 20720:20720 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9382360Z 
2022-12-01T10:48:19.9382685Z STAGE:2022-12-01 10:28:34 20721:20721 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9383066Z STAGE:2022-12-01 10:28:34 20720:20720 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9383399Z STAGE:2022-12-01 10:28:34 20720:20720 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9383706Z STAGE:2022-12-01 10:28:34 20721:20721 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9383807Z ok (7.879s)
2022-12-01T10:48:19.9383826Z 
2022-12-01T10:48:19.9384092Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9384204Z Ran 1 test in 7.879s
2022-12-01T10:48:19.9384224Z 
2022-12-01T10:48:19.9384317Z OK
2022-12-01T10:48:19.9384336Z 
2022-12-01T10:48:19.9384459Z Generating XML reports...
2022-12-01T10:48:19.9384912Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102827.xml
2022-12-01T10:48:19.9385287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9385451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9385831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9386023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9386043Z 
2022-12-01T10:48:19.9386151Z Running tests...
2022-12-01T10:48:19.9386415Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9386728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9387000Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9387219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21186
2022-12-01T10:48:19.9387438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21187
2022-12-01T10:48:19.9387840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9388024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9388406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9388597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9388958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9389130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9389934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9390138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9390368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9390623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9391041Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9391437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9391669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9391896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9392235Z STAGE:2022-12-01 10:28:43 21187:21187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9392559Z STAGE:2022-12-01 10:28:43 21186:21186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9393008Z STAGE:2022-12-01 10:28:43 21186:21186 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9393322Z STAGE:2022-12-01 10:28:43 21187:21187 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9393643Z STAGE:2022-12-01 10:28:43 21186:21186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9393956Z STAGE:2022-12-01 10:28:43 21187:21187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9394498Z STAGE:2022-12-01 10:28:45 21187:21187 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:28:45 21186:21186 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9394521Z 
2022-12-01T10:48:19.9394842Z STAGE:2022-12-01 10:28:45 21186:21186 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9395161Z STAGE:2022-12-01 10:28:45 21187:21187 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9395709Z STAGE:2022-12-01 10:28:45 21186:21186 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:28:45 21187:21187 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9395730Z 
2022-12-01T10:48:19.9395831Z ok (7.905s)
2022-12-01T10:48:19.9395850Z 
2022-12-01T10:48:19.9396116Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9396229Z Ran 1 test in 7.905s
2022-12-01T10:48:19.9396249Z 
2022-12-01T10:48:19.9396323Z OK
2022-12-01T10:48:19.9396360Z 
2022-12-01T10:48:19.9396466Z Generating XML reports...
2022-12-01T10:48:19.9396922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102838.xml
2022-12-01T10:48:19.9397295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9397473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9397914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9398106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9398126Z 
2022-12-01T10:48:19.9398229Z Running tests...
2022-12-01T10:48:19.9398497Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9398793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9399039Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:48:19.9399059Z 
2022-12-01T10:48:19.9399309Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9399419Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9399438Z 
2022-12-01T10:48:19.9399545Z OK (skipped=1)
2022-12-01T10:48:19.9399564Z 
2022-12-01T10:48:19.9399686Z Generating XML reports...
2022-12-01T10:48:19.9400141Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102848.xml
2022-12-01T10:48:19.9400513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9400689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9401046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9401238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9401257Z 
2022-12-01T10:48:19.9401365Z Running tests...
2022-12-01T10:48:19.9401625Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9401940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9402195Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:48:19.9402262Z 
2022-12-01T10:48:19.9402539Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9402652Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9402671Z 
2022-12-01T10:48:19.9402780Z OK (skipped=1)
2022-12-01T10:48:19.9402799Z 
2022-12-01T10:48:19.9402907Z Generating XML reports...
2022-12-01T10:48:19.9403359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102850.xml
2022-12-01T10:48:19.9403733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9403912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9404293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9404486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9404509Z 
2022-12-01T10:48:19.9404620Z Running tests...
2022-12-01T10:48:19.9404887Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9405188Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9405450Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:48:19.9405469Z 
2022-12-01T10:48:19.9405732Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9405844Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9405863Z 
2022-12-01T10:48:19.9405970Z OK (skipped=1)
2022-12-01T10:48:19.9405989Z 
2022-12-01T10:48:19.9406112Z Generating XML reports...
2022-12-01T10:48:19.9406562Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102852.xml
2022-12-01T10:48:19.9406930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9407167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9407543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9407737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9407757Z 
2022-12-01T10:48:19.9407865Z Running tests...
2022-12-01T10:48:19.9408129Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9408443Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9408714Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.003s)
2022-12-01T10:48:19.9408733Z 
2022-12-01T10:48:19.9408995Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9409112Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9409132Z 
2022-12-01T10:48:19.9409237Z OK (skipped=1)
2022-12-01T10:48:19.9409259Z 
2022-12-01T10:48:19.9409365Z Generating XML reports...
2022-12-01T10:48:19.9409819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102854.xml
2022-12-01T10:48:19.9410190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9410366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9410747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9410938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9410957Z 
2022-12-01T10:48:19.9411064Z Running tests...
2022-12-01T10:48:19.9411327Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9411678Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9411944Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.003s)
2022-12-01T10:48:19.9411964Z 
2022-12-01T10:48:19.9412227Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9412337Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9412357Z 
2022-12-01T10:48:19.9412463Z OK (skipped=1)
2022-12-01T10:48:19.9412482Z 
2022-12-01T10:48:19.9412603Z Generating XML reports...
2022-12-01T10:48:19.9413050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102856.xml
2022-12-01T10:48:19.9413421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9413600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9413965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9414164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9414183Z 
2022-12-01T10:48:19.9414292Z Running tests...
2022-12-01T10:48:19.9414558Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9414872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9415145Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T10:48:19.9415164Z 
2022-12-01T10:48:19.9415421Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9415532Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9415551Z 
2022-12-01T10:48:19.9415657Z OK (skipped=1)
2022-12-01T10:48:19.9415676Z 
2022-12-01T10:48:19.9415781Z Generating XML reports...
2022-12-01T10:48:19.9416288Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102858.xml
2022-12-01T10:48:19.9416681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9416856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9417234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9417426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9417445Z 
2022-12-01T10:48:19.9417553Z Running tests...
2022-12-01T10:48:19.9417815Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9418109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9418364Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T10:48:19.9418387Z 
2022-12-01T10:48:19.9418651Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9418765Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9418784Z 
2022-12-01T10:48:19.9418890Z OK (skipped=1)
2022-12-01T10:48:19.9418909Z 
2022-12-01T10:48:19.9419031Z Generating XML reports...
2022-12-01T10:48:19.9419478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102900.xml
2022-12-01T10:48:19.9419849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9420023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9420384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9420575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9420642Z 
2022-12-01T10:48:19.9420756Z Running tests...
2022-12-01T10:48:19.9421023Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9421335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9421611Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9421630Z 
2022-12-01T10:48:19.9421891Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9422003Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9422022Z 
2022-12-01T10:48:19.9422130Z OK (skipped=1)
2022-12-01T10:48:19.9422149Z 
2022-12-01T10:48:19.9422255Z Generating XML reports...
2022-12-01T10:48:19.9422706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102902.xml
2022-12-01T10:48:19.9423079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9423260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9423639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9423830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9423849Z 
2022-12-01T10:48:19.9423956Z Running tests...
2022-12-01T10:48:19.9424217Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9424531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9424797Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9424817Z 
2022-12-01T10:48:19.9425078Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9425188Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9425210Z 
2022-12-01T10:48:19.9425317Z OK (skipped=1)
2022-12-01T10:48:19.9425336Z 
2022-12-01T10:48:19.9425501Z Generating XML reports...
2022-12-01T10:48:19.9425958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102904.xml
2022-12-01T10:48:19.9426325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9426499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9426876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9427049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9427068Z 
2022-12-01T10:48:19.9427176Z Running tests...
2022-12-01T10:48:19.9427439Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9427750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9428053Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.003s)
2022-12-01T10:48:19.9428073Z 
2022-12-01T10:48:19.9428331Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9428443Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9428462Z 
2022-12-01T10:48:19.9428568Z OK (skipped=1)
2022-12-01T10:48:19.9428587Z 
2022-12-01T10:48:19.9428693Z Generating XML reports...
2022-12-01T10:48:19.9429138Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102906.xml
2022-12-01T10:48:19.9429749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9429934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9430326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9430609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9430630Z 
2022-12-01T10:48:19.9430741Z Running tests...
2022-12-01T10:48:19.9431006Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9431318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9431593Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9431632Z 
2022-12-01T10:48:19.9431876Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9431987Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9432006Z 
2022-12-01T10:48:19.9432112Z OK (skipped=1)
2022-12-01T10:48:19.9432131Z 
2022-12-01T10:48:19.9432254Z Generating XML reports...
2022-12-01T10:48:19.9432706Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102908.xml
2022-12-01T10:48:19.9433076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9433251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9433630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9433802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9433821Z 
2022-12-01T10:48:19.9433934Z Running tests...
2022-12-01T10:48:19.9434196Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9434509Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9434809Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9434833Z 
2022-12-01T10:48:19.9435152Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9435275Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9435295Z 
2022-12-01T10:48:19.9435403Z OK (skipped=1)
2022-12-01T10:48:19.9435422Z 
2022-12-01T10:48:19.9435546Z Generating XML reports...
2022-12-01T10:48:19.9435976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102910.xml
2022-12-01T10:48:19.9436343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9436516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9436894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9437087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9437110Z 
2022-12-01T10:48:19.9437218Z Running tests...
2022-12-01T10:48:19.9437483Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9437797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9438074Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9438111Z 
2022-12-01T10:48:19.9438362Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9438473Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9438492Z 
2022-12-01T10:48:19.9438597Z OK (skipped=1)
2022-12-01T10:48:19.9438616Z 
2022-12-01T10:48:19.9438739Z Generating XML reports...
2022-12-01T10:48:19.9439187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102913.xml
2022-12-01T10:48:19.9439618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9439793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9440177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9440349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9440391Z 
2022-12-01T10:48:19.9440481Z Running tests...
2022-12-01T10:48:19.9440742Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9441057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9441364Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9441384Z 
2022-12-01T10:48:19.9441650Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9441765Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9441787Z 
2022-12-01T10:48:19.9441894Z OK (skipped=1)
2022-12-01T10:48:19.9441913Z 
2022-12-01T10:48:19.9442035Z Generating XML reports...
2022-12-01T10:48:19.9442462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102915.xml
2022-12-01T10:48:19.9442831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9443007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9443381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9443572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9443592Z 
2022-12-01T10:48:19.9443700Z Running tests...
2022-12-01T10:48:19.9443961Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9444321Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9444623Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9444644Z 
2022-12-01T10:48:19.9444888Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9445000Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9445019Z 
2022-12-01T10:48:19.9445127Z OK (skipped=1)
2022-12-01T10:48:19.9445146Z 
2022-12-01T10:48:19.9445269Z Generating XML reports...
2022-12-01T10:48:19.9445718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102917.xml
2022-12-01T10:48:19.9446087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9446261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9446649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9446842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9446862Z 
2022-12-01T10:48:19.9446952Z Running tests...
2022-12-01T10:48:19.9447215Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9447530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9447831Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9447851Z 
2022-12-01T10:48:19.9448113Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9448224Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9448243Z 
2022-12-01T10:48:19.9448349Z OK (skipped=1)
2022-12-01T10:48:19.9448413Z 
2022-12-01T10:48:19.9448544Z Generating XML reports...
2022-12-01T10:48:19.9448997Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102919.xml
2022-12-01T10:48:19.9449348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9449523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9449901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9450092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9450112Z 
2022-12-01T10:48:19.9450219Z Running tests...
2022-12-01T10:48:19.9450482Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9450794Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9451087Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9451107Z 
2022-12-01T10:48:19.9451369Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9451462Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9451481Z 
2022-12-01T10:48:19.9451586Z OK (skipped=1)
2022-12-01T10:48:19.9451605Z 
2022-12-01T10:48:19.9451729Z Generating XML reports...
2022-12-01T10:48:19.9452174Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102921.xml
2022-12-01T10:48:19.9452541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9452715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9453091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9453330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9453351Z 
2022-12-01T10:48:19.9453446Z Running tests...
2022-12-01T10:48:19.9453709Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9454020Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9454319Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9454338Z 
2022-12-01T10:48:19.9454595Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9454706Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9454726Z 
2022-12-01T10:48:19.9454830Z OK (skipped=1)
2022-12-01T10:48:19.9454850Z 
2022-12-01T10:48:19.9454971Z Generating XML reports...
2022-12-01T10:48:19.9455423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102923.xml
2022-12-01T10:48:19.9455789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9455966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9456347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9456541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9456563Z 
2022-12-01T10:48:19.9456672Z Running tests...
2022-12-01T10:48:19.9456933Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9457244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9457538Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9457603Z 
2022-12-01T10:48:19.9457873Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9457968Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9457986Z 
2022-12-01T10:48:19.9458095Z OK (skipped=1)
2022-12-01T10:48:19.9458113Z 
2022-12-01T10:48:19.9458236Z Generating XML reports...
2022-12-01T10:48:19.9458682Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102925.xml
2022-12-01T10:48:19.9459052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9459229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9459607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9459799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9459818Z 
2022-12-01T10:48:19.9459931Z Running tests...
2022-12-01T10:48:19.9460183Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9460500Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9460806Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9460827Z 
2022-12-01T10:48:19.9461087Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9461198Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9461217Z 
2022-12-01T10:48:19.9461322Z OK (skipped=1)
2022-12-01T10:48:19.9461341Z 
2022-12-01T10:48:19.9461461Z Generating XML reports...
2022-12-01T10:48:19.9461904Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102927.xml
2022-12-01T10:48:19.9462254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9462487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9462878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9463069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9463088Z 
2022-12-01T10:48:19.9463194Z Running tests...
2022-12-01T10:48:19.9463453Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9463763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9464063Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9464082Z 
2022-12-01T10:48:19.9464341Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9464435Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9464477Z 
2022-12-01T10:48:19.9464567Z OK (skipped=1)
2022-12-01T10:48:19.9464585Z 
2022-12-01T10:48:19.9464712Z Generating XML reports...
2022-12-01T10:48:19.9465160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102929.xml
2022-12-01T10:48:19.9465533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9465709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9466089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9466281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9466300Z 
2022-12-01T10:48:19.9466409Z Running tests...
2022-12-01T10:48:19.9466653Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9467025Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9467339Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9467359Z 
2022-12-01T10:48:19.9467624Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9467735Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9467754Z 
2022-12-01T10:48:19.9467861Z OK (skipped=1)
2022-12-01T10:48:19.9467880Z 
2022-12-01T10:48:19.9468004Z Generating XML reports...
2022-12-01T10:48:19.9468452Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102931.xml
2022-12-01T10:48:19.9468821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9468981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9469586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9469795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9469816Z 
2022-12-01T10:48:19.9469925Z Running tests...
2022-12-01T10:48:19.9470195Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9470507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9470802Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T10:48:19.9470822Z 
2022-12-01T10:48:19.9471080Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9471189Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9471209Z 
2022-12-01T10:48:19.9471297Z OK (skipped=1)
2022-12-01T10:48:19.9471316Z 
2022-12-01T10:48:19.9471438Z Generating XML reports...
2022-12-01T10:48:19.9471977Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102933.xml
2022-12-01T10:48:19.9472364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9472538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9472918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9473108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9473128Z 
2022-12-01T10:48:19.9473237Z Running tests...
2022-12-01T10:48:19.9473483Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9473796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9474098Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T10:48:19.9474126Z 
2022-12-01T10:48:19.9474387Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9474497Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9474516Z 
2022-12-01T10:48:19.9474623Z OK (skipped=1)
2022-12-01T10:48:19.9474642Z 
2022-12-01T10:48:19.9474766Z Generating XML reports...
2022-12-01T10:48:19.9475209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102936.xml
2022-12-01T10:48:19.9475578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9475734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9476113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9476302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9476384Z 
2022-12-01T10:48:19.9476505Z Running tests...
2022-12-01T10:48:19.9476771Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9477084Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9477347Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9477566Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23838
2022-12-01T10:48:19.9477786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23839
2022-12-01T10:48:19.9478139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9478318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9478749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9478946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9479312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9479487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9479858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9480046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9480275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9480519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9480921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9481372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9481611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9481838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9482076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9482316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9482719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9483098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9483209Z ok (6.898s)
2022-12-01T10:48:19.9483229Z 
2022-12-01T10:48:19.9483498Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9483612Z Ran 1 test in 6.899s
2022-12-01T10:48:19.9483631Z 
2022-12-01T10:48:19.9483724Z OK
2022-12-01T10:48:19.9483743Z 
2022-12-01T10:48:19.9483867Z Generating XML reports...
2022-12-01T10:48:19.9484317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102937.xml
2022-12-01T10:48:19.9484691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9484868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9485232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9485423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9485489Z 
2022-12-01T10:48:19.9485604Z Running tests...
2022-12-01T10:48:19.9485873Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9486191Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9486451Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9486668Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24240
2022-12-01T10:48:19.9486886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24241
2022-12-01T10:48:19.9487240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9487417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9487798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9487994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9488362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9488535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9488912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9489103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9489349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9489576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9489978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9490375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9490651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9490893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9491043Z skip: Need at least 3 CUDA devices (3.738s)
2022-12-01T10:48:19.9491063Z 
2022-12-01T10:48:19.9491333Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9491445Z Ran 1 test in 3.739s
2022-12-01T10:48:19.9491464Z 
2022-12-01T10:48:19.9491572Z OK (skipped=1)
2022-12-01T10:48:19.9491592Z 
2022-12-01T10:48:19.9491698Z Generating XML reports...
2022-12-01T10:48:19.9492152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102946.xml
2022-12-01T10:48:19.9492521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9492703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9493080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9493268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9493288Z 
2022-12-01T10:48:19.9493396Z Running tests...
2022-12-01T10:48:19.9493659Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9493953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9494204Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-12-01T10:48:19.9494224Z 
2022-12-01T10:48:19.9494486Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9494598Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9494663Z 
2022-12-01T10:48:19.9494776Z OK (skipped=1)
2022-12-01T10:48:19.9494795Z 
2022-12-01T10:48:19.9494923Z Generating XML reports...
2022-12-01T10:48:19.9495372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102952.xml
2022-12-01T10:48:19.9495741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9495915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9496272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9496463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9496482Z 
2022-12-01T10:48:19.9496590Z Running tests...
2022-12-01T10:48:19.9496850Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9497162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9497412Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9497632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24599
2022-12-01T10:48:19.9497849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24600
2022-12-01T10:48:19.9498203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9498381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9498759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9498949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9499315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9499496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9499913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9500112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9500360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9500587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9500992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9501388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9501622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9501860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9501963Z ok (4.786s)
2022-12-01T10:48:19.9501982Z 
2022-12-01T10:48:19.9502247Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9502359Z Ran 1 test in 4.786s
2022-12-01T10:48:19.9502378Z 
2022-12-01T10:48:19.9502471Z OK
2022-12-01T10:48:19.9502489Z 
2022-12-01T10:48:19.9502595Z Generating XML reports...
2022-12-01T10:48:19.9503047Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102955.xml
2022-12-01T10:48:19.9503418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9503594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9503971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9504217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9504239Z 
2022-12-01T10:48:19.9504351Z Running tests...
2022-12-01T10:48:19.9504616Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9504912Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9505164Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9505384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24992
2022-12-01T10:48:19.9505599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24993
2022-12-01T10:48:19.9505970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9506145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9506524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9506720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9507088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9507243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9507615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9507804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9508049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9508293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9508694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9509136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9509746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9509966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9510071Z ok (6.497s)
2022-12-01T10:48:19.9510090Z 
2022-12-01T10:48:19.9510369Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9510482Z Ran 1 test in 6.497s
2022-12-01T10:48:19.9510501Z 
2022-12-01T10:48:19.9510593Z OK
2022-12-01T10:48:19.9510612Z 
2022-12-01T10:48:19.9510737Z Generating XML reports...
2022-12-01T10:48:19.9511188Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103002.xml
2022-12-01T10:48:19.9511568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9511746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9512106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9512296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9512315Z 
2022-12-01T10:48:19.9512423Z Running tests...
2022-12-01T10:48:19.9512683Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9512994Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9513253Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9513473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25301
2022-12-01T10:48:19.9513776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25302
2022-12-01T10:48:19.9514139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9514316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9514695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9514887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9515251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9515424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9515794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9515984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9516305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9516533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9516934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9517329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9517560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9517788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9518027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9518268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9518761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9519170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9519256Z ok (4.931s)
2022-12-01T10:48:19.9519276Z 
2022-12-01T10:48:19.9519540Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9519657Z Ran 1 test in 4.932s
2022-12-01T10:48:19.9519676Z 
2022-12-01T10:48:19.9519769Z OK
2022-12-01T10:48:19.9519788Z 
2022-12-01T10:48:19.9519912Z Generating XML reports...
2022-12-01T10:48:19.9520360Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103010.xml
2022-12-01T10:48:19.9520731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9520915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9521281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9521475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9521494Z 
2022-12-01T10:48:19.9521605Z Running tests...
2022-12-01T10:48:19.9521868Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9522184Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9522453Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9522673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25594
2022-12-01T10:48:19.9522891Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25595
2022-12-01T10:48:19.9523326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9523486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9529114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9529320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9529695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9529876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9530259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9530452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9530702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9530964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9531351Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9531752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9531985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9532216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9532378Z skip: Skipped due to small world size. (3.945s)
2022-12-01T10:48:19.9532399Z 
2022-12-01T10:48:19.9532667Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9532784Z Ran 1 test in 3.945s
2022-12-01T10:48:19.9532807Z 
2022-12-01T10:48:19.9532919Z OK (skipped=1)
2022-12-01T10:48:19.9532938Z 
2022-12-01T10:48:19.9533164Z Generating XML reports...
2022-12-01T10:48:19.9533619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103018.xml
2022-12-01T10:48:19.9533998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9534180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9534565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9534762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9534782Z 
2022-12-01T10:48:19.9534894Z Running tests...
2022-12-01T10:48:19.9535161Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9535482Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9535730Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9535955Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25914
2022-12-01T10:48:19.9536173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25915
2022-12-01T10:48:19.9536548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9536725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9537106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9537300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9537664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9537898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9538256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9538448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9538695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9538940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9539342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9539742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9539976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9540214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9540357Z skip: Skipped due to small world size. (3.969s)
2022-12-01T10:48:19.9540396Z 
2022-12-01T10:48:19.9540647Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9540765Z Ran 1 test in 3.969s
2022-12-01T10:48:19.9540785Z 
2022-12-01T10:48:19.9540896Z OK (skipped=1)
2022-12-01T10:48:19.9540916Z 
2022-12-01T10:48:19.9541042Z Generating XML reports...
2022-12-01T10:48:19.9541497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103024.xml
2022-12-01T10:48:19.9541870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9542050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9542428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9542652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9542695Z 
2022-12-01T10:48:19.9542791Z Running tests...
2022-12-01T10:48:19.9543060Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9543379Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9543644Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9543866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26205
2022-12-01T10:48:19.9544085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26206
2022-12-01T10:48:19.9544460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9544641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9545016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9545213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9545586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9545760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9546139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9546331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9546576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9546820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9547266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9547669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9547900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9548131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9548292Z skip: Skipped due to small world size. (3.958s)
2022-12-01T10:48:19.9548313Z 
2022-12-01T10:48:19.9548582Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9548697Z Ran 1 test in 3.958s
2022-12-01T10:48:19.9548717Z 
2022-12-01T10:48:19.9548826Z OK (skipped=1)
2022-12-01T10:48:19.9548845Z 
2022-12-01T10:48:19.9548971Z Generating XML reports...
2022-12-01T10:48:19.9549673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103030.xml
2022-12-01T10:48:19.9550086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9550268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9550650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9550845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9550866Z 
2022-12-01T10:48:19.9550976Z Running tests...
2022-12-01T10:48:19.9551242Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9551560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9551833Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9552131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26441
2022-12-01T10:48:19.9552365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26442
2022-12-01T10:48:19.9552744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9552923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9553300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9553491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9553859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9554037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9554394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9554593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9554841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9555086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9555487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9555888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9556118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9556345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9556591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9556891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9557295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9557690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9557800Z ok (5.198s)
2022-12-01T10:48:19.9557821Z 
2022-12-01T10:48:19.9558087Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9558202Z Ran 1 test in 5.198s
2022-12-01T10:48:19.9558222Z 
2022-12-01T10:48:19.9558318Z OK
2022-12-01T10:48:19.9558338Z 
2022-12-01T10:48:19.9558465Z Generating XML reports...
2022-12-01T10:48:19.9558918Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103036.xml
2022-12-01T10:48:19.9559282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9559464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9559846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9560039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9560059Z 
2022-12-01T10:48:19.9560168Z Running tests...
2022-12-01T10:48:19.9560432Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9560748Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9561159Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2022-12-01T10:48:19.9561184Z 
2022-12-01T10:48:19.9561498Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9561602Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9561622Z 
2022-12-01T10:48:19.9561732Z OK (skipped=1)
2022-12-01T10:48:19.9561752Z 
2022-12-01T10:48:19.9561877Z Generating XML reports...
2022-12-01T10:48:19.9562331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103044.xml
2022-12-01T10:48:19.9562705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9562885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9563262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9563454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9563478Z 
2022-12-01T10:48:19.9563588Z Running tests...
2022-12-01T10:48:19.9563840Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9564154Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9564420Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9564641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26851
2022-12-01T10:48:19.9564861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26852
2022-12-01T10:48:19.9565237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9565415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9565795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9566026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9566407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9566585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9566961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9567152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9567399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9567644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9568043Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9568451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9568666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9568896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9569058Z skip: Skipped due to small world size. (4.038s)
2022-12-01T10:48:19.9569078Z 
2022-12-01T10:48:19.9569345Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9569461Z Ran 1 test in 4.038s
2022-12-01T10:48:19.9569481Z 
2022-12-01T10:48:19.9569590Z OK (skipped=1)
2022-12-01T10:48:19.9569609Z 
2022-12-01T10:48:19.9569735Z Generating XML reports...
2022-12-01T10:48:19.9570191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103046.xml
2022-12-01T10:48:19.9570545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9570778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9571173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9571364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9571384Z 
2022-12-01T10:48:19.9571495Z Running tests...
2022-12-01T10:48:19.9571758Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9572075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9572345Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9572569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27163
2022-12-01T10:48:19.9572768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27164
2022-12-01T10:48:19.9573151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9573330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9573714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9573907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9574270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9574447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9574819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9574990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9575297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9575541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9575946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9576343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9576577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9576806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9576911Z ok (3.935s)
2022-12-01T10:48:19.9576931Z 
2022-12-01T10:48:19.9577197Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9577294Z Ran 1 test in 3.935s
2022-12-01T10:48:19.9577318Z 
2022-12-01T10:48:19.9577415Z OK
2022-12-01T10:48:19.9577434Z 
2022-12-01T10:48:19.9577563Z Generating XML reports...
2022-12-01T10:48:19.9578066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103052.xml
2022-12-01T10:48:19.9578444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9578622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9579004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9579197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9579216Z 
2022-12-01T10:48:19.9579327Z Running tests...
2022-12-01T10:48:19.9579577Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9579892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9580223Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9580455Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27406
2022-12-01T10:48:19.9580675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27407
2022-12-01T10:48:19.9581052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9581230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9581609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9581784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9582149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9582333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9582710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9582901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9583147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9583394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9583795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9584194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9584407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9584694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9584800Z ok (3.960s)
2022-12-01T10:48:19.9584821Z 
2022-12-01T10:48:19.9585091Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9585208Z Ran 1 test in 3.960s
2022-12-01T10:48:19.9585228Z 
2022-12-01T10:48:19.9585323Z OK
2022-12-01T10:48:19.9585343Z 
2022-12-01T10:48:19.9585469Z Generating XML reports...
2022-12-01T10:48:19.9585927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103058.xml
2022-12-01T10:48:19.9586286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9586468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9586851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9587051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9587071Z 
2022-12-01T10:48:19.9587182Z Running tests...
2022-12-01T10:48:19.9587448Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9587766Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9588048Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9588067Z 
2022-12-01T10:48:19.9588330Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9588427Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9588447Z 
2022-12-01T10:48:19.9588559Z OK (skipped=1)
2022-12-01T10:48:19.9588578Z 
2022-12-01T10:48:19.9588703Z Generating XML reports...
2022-12-01T10:48:19.9589154Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103104.xml
2022-12-01T10:48:19.9589929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9590130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9590524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9590717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9590739Z 
2022-12-01T10:48:19.9590850Z Running tests...
2022-12-01T10:48:19.9591100Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9591417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9591678Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-12-01T10:48:19.9591697Z 
2022-12-01T10:48:19.9591965Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9592083Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9592103Z 
2022-12-01T10:48:19.9592212Z OK (skipped=1)
2022-12-01T10:48:19.9592231Z 
2022-12-01T10:48:19.9592358Z Generating XML reports...
2022-12-01T10:48:19.9592814Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103106.xml
2022-12-01T10:48:19.9593192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9593350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9593733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9593929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9593948Z 
2022-12-01T10:48:19.9594058Z Running tests...
2022-12-01T10:48:19.9594401Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9594724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9595000Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9595021Z 
2022-12-01T10:48:19.9595285Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9595381Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9595420Z 
2022-12-01T10:48:19.9595512Z OK (skipped=1)
2022-12-01T10:48:19.9595531Z 
2022-12-01T10:48:19.9595655Z Generating XML reports...
2022-12-01T10:48:19.9596104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103109.xml
2022-12-01T10:48:19.9596475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9596656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9597039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9597234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9597253Z 
2022-12-01T10:48:19.9597363Z Running tests...
2022-12-01T10:48:19.9597611Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9597930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9598192Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9598212Z 
2022-12-01T10:48:19.9598471Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9598584Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9598603Z 
2022-12-01T10:48:19.9598712Z OK (skipped=1)
2022-12-01T10:48:19.9598735Z 
2022-12-01T10:48:19.9598864Z Generating XML reports...
2022-12-01T10:48:19.9599378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103111.xml
2022-12-01T10:48:19.9599768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9599928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9600310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9600509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9600528Z 
2022-12-01T10:48:19.9600638Z Running tests...
2022-12-01T10:48:19.9600905Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9601219Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9601491Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9601511Z 
2022-12-01T10:48:19.9601772Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9601868Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9601905Z 
2022-12-01T10:48:19.9601997Z OK (skipped=1)
2022-12-01T10:48:19.9602016Z 
2022-12-01T10:48:19.9602142Z Generating XML reports...
2022-12-01T10:48:19.9602591Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103113.xml
2022-12-01T10:48:19.9602968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9603147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9603523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9603788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9603811Z 
2022-12-01T10:48:19.9603924Z Running tests...
2022-12-01T10:48:19.9604176Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9604489Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9604768Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9604788Z 
2022-12-01T10:48:19.9605053Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9605166Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9605185Z 
2022-12-01T10:48:19.9605294Z OK (skipped=1)
2022-12-01T10:48:19.9605313Z 
2022-12-01T10:48:19.9605438Z Generating XML reports...
2022-12-01T10:48:19.9605887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103115.xml
2022-12-01T10:48:19.9606267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9606429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9606807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9607002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9607021Z 
2022-12-01T10:48:19.9607131Z Running tests...
2022-12-01T10:48:19.9607396Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9607713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9607978Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9607998Z 
2022-12-01T10:48:19.9608261Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9608379Z Ran 1 test in 0.003s
2022-12-01T10:48:19.9608455Z 
2022-12-01T10:48:19.9608551Z OK (skipped=1)
2022-12-01T10:48:19.9608570Z 
2022-12-01T10:48:19.9608698Z Generating XML reports...
2022-12-01T10:48:19.9609148Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103117.xml
2022-12-01T10:48:19.9609518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9609695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9610071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9610264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9610284Z 
2022-12-01T10:48:19.9610394Z Running tests...
2022-12-01T10:48:19.9610639Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9610963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9611229Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T10:48:19.9611250Z 
2022-12-01T10:48:19.9611516Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9611630Z Ran 1 test in 0.002s
2022-12-01T10:48:19.9611650Z 
2022-12-01T10:48:19.9611759Z OK (skipped=1)
2022-12-01T10:48:19.9611778Z 
2022-12-01T10:48:19.9611903Z Generating XML reports...
2022-12-01T10:48:19.9612351Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103119.xml
2022-12-01T10:48:19.9612728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9612885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9613328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9613522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9613542Z 
2022-12-01T10:48:19.9613652Z Running tests...
2022-12-01T10:48:19.9613915Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9614229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9614476Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9614695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28303
2022-12-01T10:48:19.9614895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28304
2022-12-01T10:48:19.9615270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9615455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9615838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9616031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9616392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9616568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9616949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9617138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9617369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9617624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9618077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9618488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9618722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9619069Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9619299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9619634Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9619972Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9620300Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9620630Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9620950Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9621279Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9621606Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9621936Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9622262Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9622597Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9622993Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9623301Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9623633Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9623972Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9624299Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9624624Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9624943Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9625276Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9625615Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9625945Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9626250Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9626583Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9626915Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9627242Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9627562Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9627891Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9628259Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9628601Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9628925Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9629239Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9629766Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9630099Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9630425Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9630759Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9631096Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9631426Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9631748Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9632066Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9632397Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9632731Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9633059Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9633391Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9633818Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9634150Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9634469Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9634801Z STAGE:2022-12-01 10:31:25 28304:28304 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9635116Z STAGE:2022-12-01 10:31:25 28303:28303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9635223Z ok (4.108s)
2022-12-01T10:48:19.9635244Z 
2022-12-01T10:48:19.9635509Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9635626Z Ran 1 test in 4.108s
2022-12-01T10:48:19.9635646Z 
2022-12-01T10:48:19.9635742Z OK
2022-12-01T10:48:19.9635765Z 
2022-12-01T10:48:19.9635894Z Generating XML reports...
2022-12-01T10:48:19.9636356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103121.xml
2022-12-01T10:48:19.9636732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9636911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9637274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9637470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9637490Z 
2022-12-01T10:48:19.9637600Z Running tests...
2022-12-01T10:48:19.9637867Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9638185Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9638517Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9639290Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s)
2022-12-01T10:48:19.9639311Z 
2022-12-01T10:48:19.9639574Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9639688Z Ran 1 test in 1.668s
2022-12-01T10:48:19.9639708Z 
2022-12-01T10:48:19.9639799Z OK (skipped=1)
2022-12-01T10:48:19.9639837Z 
2022-12-01T10:48:19.9639945Z Generating XML reports...
2022-12-01T10:48:19.9640397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103127.xml
2022-12-01T10:48:19.9640774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9640961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9641345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9641539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9641558Z 
2022-12-01T10:48:19.9641669Z Running tests...
2022-12-01T10:48:19.9641937Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9642235Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9642500Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9642721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28766
2022-12-01T10:48:19.9642942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28767
2022-12-01T10:48:19.9643383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9643562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9643943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9644135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9644484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9644665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9645042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9645236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9645491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9645739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9646144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9646545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9646777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9646988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9647233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9647475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9647926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9648338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9648675Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9649010Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9649559Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9649580Z 
2022-12-01T10:48:19.9649910Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9650233Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9650562Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9650896Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9651223Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9651546Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9651876Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9652211Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9652539Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9652923Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9653258Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9653568Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9653896Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9654219Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9654553Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9654880Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9655205Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9655536Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9655872Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9656206Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9656514Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9656835Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9657169Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9657498Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9657823Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9658193Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9658565Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9658897Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9659224Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9659529Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9659860Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9660188Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9660512Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9660841Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9661178Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9661502Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9661826Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9662150Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9662461Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9662783Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9663108Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9663532Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:19.9663866Z STAGE:2022-12-01 10:31:35 28767:28767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9664197Z STAGE:2022-12-01 10:31:35 28766:28766 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:19.9664301Z ok (3.938s)
2022-12-01T10:48:19.9664322Z 
2022-12-01T10:48:19.9664589Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9664686Z Ran 1 test in 3.939s
2022-12-01T10:48:19.9664726Z 
2022-12-01T10:48:19.9664803Z OK
2022-12-01T10:48:19.9664822Z 
2022-12-01T10:48:19.9664947Z Generating XML reports...
2022-12-01T10:48:19.9665400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103131.xml
2022-12-01T10:48:19.9665772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9665958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9666341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9666537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9666557Z 
2022-12-01T10:48:19.9666672Z Running tests...
2022-12-01T10:48:19.9666920Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9667237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9667494Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9667717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29050
2022-12-01T10:48:19.9667938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29051
2022-12-01T10:48:19.9668371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9668559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9668946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9669120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9669678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9669861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9670242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9670431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9670687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9670936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9671340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9671743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9671960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9672196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9672359Z skip: Skipped due to small world size. (3.856s)
2022-12-01T10:48:19.9672380Z 
2022-12-01T10:48:19.9672652Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9672767Z Ran 1 test in 3.856s
2022-12-01T10:48:19.9672863Z 
2022-12-01T10:48:19.9672984Z OK (skipped=1)
2022-12-01T10:48:19.9673003Z 
2022-12-01T10:48:19.9673134Z Generating XML reports...
2022-12-01T10:48:19.9673592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103137.xml
2022-12-01T10:48:19.9673965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9674126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9674507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9674702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9674722Z 
2022-12-01T10:48:19.9674836Z Running tests...
2022-12-01T10:48:19.9675103Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9675421Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9675692Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9675911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29336
2022-12-01T10:48:19.9676112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29337
2022-12-01T10:48:19.9676488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9676666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9677050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9677243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9677605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9677845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9678287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9678479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9678708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9678954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9679359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9679753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9679987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9680225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9681000Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:48:19.9681118Z   warnings.warn(
2022-12-01T10:48:19.9681875Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T10:48:19.9681990Z   warnings.warn(
2022-12-01T10:48:19.9682076Z ok (5.737s)
2022-12-01T10:48:19.9682096Z 
2022-12-01T10:48:19.9682364Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9682536Z Ran 1 test in 5.738s
2022-12-01T10:48:19.9682559Z 
2022-12-01T10:48:19.9682656Z OK
2022-12-01T10:48:19.9682675Z 
2022-12-01T10:48:19.9682803Z Generating XML reports...
2022-12-01T10:48:19.9683264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103144.xml
2022-12-01T10:48:19.9683641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9683820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9684182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9684375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9684394Z 
2022-12-01T10:48:19.9684504Z Running tests...
2022-12-01T10:48:19.9684771Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9685096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9685365Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9686114Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.685s)
2022-12-01T10:48:19.9686135Z 
2022-12-01T10:48:19.9686401Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9686516Z Ran 1 test in 1.685s
2022-12-01T10:48:19.9686535Z 
2022-12-01T10:48:19.9686645Z OK (skipped=1)
2022-12-01T10:48:19.9686665Z 
2022-12-01T10:48:19.9686773Z Generating XML reports...
2022-12-01T10:48:19.9687226Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103151.xml
2022-12-01T10:48:19.9687650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9687838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9688224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9688417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9688438Z 
2022-12-01T10:48:19.9688548Z Running tests...
2022-12-01T10:48:19.9688815Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9689114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9689430Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9690187Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.780s)
2022-12-01T10:48:19.9690209Z 
2022-12-01T10:48:19.9690472Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9690588Z Ran 1 test in 1.780s
2022-12-01T10:48:19.9690608Z 
2022-12-01T10:48:19.9690717Z OK (skipped=1)
2022-12-01T10:48:19.9690737Z 
2022-12-01T10:48:19.9690866Z Generating XML reports...
2022-12-01T10:48:19.9691316Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103155.xml
2022-12-01T10:48:19.9691689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9691925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9692295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9692490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9692510Z 
2022-12-01T10:48:19.9692620Z Running tests...
2022-12-01T10:48:19.9692884Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9693200Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9693521Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9694266Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.678s)
2022-12-01T10:48:19.9694289Z 
2022-12-01T10:48:19.9694552Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9694667Z Ran 1 test in 1.678s
2022-12-01T10:48:19.9694687Z 
2022-12-01T10:48:19.9694796Z OK (skipped=1)
2022-12-01T10:48:19.9694816Z 
2022-12-01T10:48:19.9694924Z Generating XML reports...
2022-12-01T10:48:19.9695375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103159.xml
2022-12-01T10:48:19.9695750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9695929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9696310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9696510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9696574Z 
2022-12-01T10:48:19.9696692Z Running tests...
2022-12-01T10:48:19.9696961Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9697257Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9697525Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9697746Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30068
2022-12-01T10:48:19.9697966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30069
2022-12-01T10:48:19.9698340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9698520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9698906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9699106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9699475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9699636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9700023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9700217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9700464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9700868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9701110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9701565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9701800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9702036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9702278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12ou5ymq
2022-12-01T10:48:19.9702553Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12ou5ymq/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9702812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y22aptt
2022-12-01T10:48:19.9703084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y22aptt/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9703324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9703570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9703675Z ok (6.145s)
2022-12-01T10:48:19.9703695Z 
2022-12-01T10:48:19.9703966Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9704062Z Ran 1 test in 6.145s
2022-12-01T10:48:19.9704082Z 
2022-12-01T10:48:19.9704179Z OK
2022-12-01T10:48:19.9704198Z 
2022-12-01T10:48:19.9704323Z Generating XML reports...
2022-12-01T10:48:19.9704775Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103203.xml
2022-12-01T10:48:19.9705148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9705331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9705713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9705957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9705979Z 
2022-12-01T10:48:19.9706098Z Running tests...
2022-12-01T10:48:19.9706349Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9706669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9706949Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9707173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30419
2022-12-01T10:48:19.9707394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30420
2022-12-01T10:48:19.9707763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9707943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9708335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9708511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9708883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9709061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9709610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9709811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9710061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9710308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9710818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9711225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9711441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9711675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9711935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k5yqpoq
2022-12-01T10:48:19.9712209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k5yqpoq/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9712463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n2plncu
2022-12-01T10:48:19.9712733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n2plncu/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9712978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9713217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9713433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9713666Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9713771Z ok (6.124s)
2022-12-01T10:48:19.9713793Z 
2022-12-01T10:48:19.9714064Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9714179Z Ran 1 test in 6.124s
2022-12-01T10:48:19.9714199Z 
2022-12-01T10:48:19.9714294Z OK
2022-12-01T10:48:19.9714313Z 
2022-12-01T10:48:19.9714439Z Generating XML reports...
2022-12-01T10:48:19.9714894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103212.xml
2022-12-01T10:48:19.9715330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9715506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9715895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9716090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9716110Z 
2022-12-01T10:48:19.9716222Z Running tests...
2022-12-01T10:48:19.9716484Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9716804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9717077Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9717831Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.735s)
2022-12-01T10:48:19.9717855Z 
2022-12-01T10:48:19.9718126Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9718222Z Ran 1 test in 1.736s
2022-12-01T10:48:19.9718261Z 
2022-12-01T10:48:19.9718352Z OK (skipped=1)
2022-12-01T10:48:19.9718371Z 
2022-12-01T10:48:19.9718498Z Generating XML reports...
2022-12-01T10:48:19.9718949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103220.xml
2022-12-01T10:48:19.9719323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9719501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9719882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9720133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9720153Z 
2022-12-01T10:48:19.9720266Z Running tests...
2022-12-01T10:48:19.9720515Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9720830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9721125Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9721871Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.725s)
2022-12-01T10:48:19.9721891Z 
2022-12-01T10:48:19.9722162Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9722278Z Ran 1 test in 1.725s
2022-12-01T10:48:19.9722298Z 
2022-12-01T10:48:19.9722408Z OK (skipped=1)
2022-12-01T10:48:19.9722428Z 
2022-12-01T10:48:19.9722553Z Generating XML reports...
2022-12-01T10:48:19.9723003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103224.xml
2022-12-01T10:48:19.9723380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9723541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9723930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9724125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9724144Z 
2022-12-01T10:48:19.9724255Z Running tests...
2022-12-01T10:48:19.9724527Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9724890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9725189Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9725410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31061
2022-12-01T10:48:19.9725612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31062
2022-12-01T10:48:19.9725992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9726171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9726549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9726745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9727117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9727293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9727667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9727861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9728090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9728338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9728744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9729142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9729433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9729665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9729928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz2jnp9hp
2022-12-01T10:48:19.9730201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz2jnp9hp/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9730459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph8ddizj_
2022-12-01T10:48:19.9730709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph8ddizj_/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9730930Z <class 'torch.nn.parameter.Parameter'>
2022-12-01T10:48:19.9731039Z ok (5.459s)
2022-12-01T10:48:19.9731059Z 
2022-12-01T10:48:19.9731327Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9731447Z Ran 1 test in 5.460s
2022-12-01T10:48:19.9731470Z 
2022-12-01T10:48:19.9731565Z OK
2022-12-01T10:48:19.9731584Z 
2022-12-01T10:48:19.9731711Z Generating XML reports...
2022-12-01T10:48:19.9732164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103228.xml
2022-12-01T10:48:19.9732522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9732701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9733082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9733275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9733294Z 
2022-12-01T10:48:19.9733404Z Running tests...
2022-12-01T10:48:19.9733669Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9734048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9734369Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9734589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31357
2022-12-01T10:48:19.9734791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31358
2022-12-01T10:48:19.9735167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9735345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9735725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9735919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9736293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9736472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9736846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9737020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9737270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9737514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9737918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9738309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9738599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9738831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9739088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_d6xcxjx
2022-12-01T10:48:19.9739360Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_d6xcxjx/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9739602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppy8vw3zh
2022-12-01T10:48:19.9739872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppy8vw3zh/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9739979Z ok (5.665s)
2022-12-01T10:48:19.9739999Z 
2022-12-01T10:48:19.9740274Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9740390Z Ran 1 test in 5.666s
2022-12-01T10:48:19.9740415Z 
2022-12-01T10:48:19.9740511Z OK
2022-12-01T10:48:19.9740531Z 
2022-12-01T10:48:19.9740658Z Generating XML reports...
2022-12-01T10:48:19.9741107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103236.xml
2022-12-01T10:48:19.9741462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9741642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9742024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9742218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9742238Z 
2022-12-01T10:48:19.9742349Z Running tests...
2022-12-01T10:48:19.9742615Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9742932Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9743245Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9743475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31654
2022-12-01T10:48:19.9743677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31655
2022-12-01T10:48:19.9744056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9744235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9744619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9744819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9745185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9745367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9745752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9745928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9746174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9746416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9746816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9747216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9747447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9747736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9747997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdl7smrji
2022-12-01T10:48:19.9748270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdl7smrji/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9748507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp794vhrzt
2022-12-01T10:48:19.9748778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp794vhrzt/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9749017Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9749254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9749587Z ok (6.096s)
2022-12-01T10:48:19.9749613Z 
2022-12-01T10:48:19.9749907Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9750032Z Ran 1 test in 6.096s
2022-12-01T10:48:19.9750053Z 
2022-12-01T10:48:19.9750154Z OK
2022-12-01T10:48:19.9750173Z 
2022-12-01T10:48:19.9750300Z Generating XML reports...
2022-12-01T10:48:19.9750742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103244.xml
2022-12-01T10:48:19.9751118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9751299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9751681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9751877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9751897Z 
2022-12-01T10:48:19.9752008Z Running tests...
2022-12-01T10:48:19.9752269Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9752662Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9752946Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9753167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31988
2022-12-01T10:48:19.9753386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31989
2022-12-01T10:48:19.9753762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9753940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9754320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9754514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9754876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9755059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9755417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9755612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9755859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9756107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9756511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9756915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9757227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9757462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9757702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnte_lra
2022-12-01T10:48:19.9757975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnte_lra/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9758232Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu511cbr6
2022-12-01T10:48:19.9758501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu511cbr6/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9759280Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:19.9759391Z ok (6.224s)
2022-12-01T10:48:19.9759411Z 
2022-12-01T10:48:19.9759685Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9759801Z Ran 1 test in 6.224s
2022-12-01T10:48:19.9759821Z 
2022-12-01T10:48:19.9759918Z OK
2022-12-01T10:48:19.9759937Z 
2022-12-01T10:48:19.9760063Z Generating XML reports...
2022-12-01T10:48:19.9760518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103252.xml
2022-12-01T10:48:19.9760874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9761054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9761489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9761691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9761711Z 
2022-12-01T10:48:19.9761822Z Running tests...
2022-12-01T10:48:19.9762091Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9762409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9762692Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9763440Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.721s)
2022-12-01T10:48:19.9763466Z 
2022-12-01T10:48:19.9763736Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9763835Z Ran 1 test in 1.721s
2022-12-01T10:48:19.9763854Z 
2022-12-01T10:48:19.9763967Z OK (skipped=1)
2022-12-01T10:48:19.9763986Z 
2022-12-01T10:48:19.9764113Z Generating XML reports...
2022-12-01T10:48:19.9764564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103301.xml
2022-12-01T10:48:19.9764941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9765122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9765503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9765696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9765761Z 
2022-12-01T10:48:19.9765859Z Running tests...
2022-12-01T10:48:19.9766132Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9766450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9766708Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9766930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32406
2022-12-01T10:48:19.9767151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32407
2022-12-01T10:48:19.9767519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9767697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9768059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9768262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9768632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9768810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9769183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9769375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9769626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9769874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9770276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9770657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9770943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9771182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9771443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx37x_a4v
2022-12-01T10:48:19.9771701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsf_bmi5
2022-12-01T10:48:19.9771969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx37x_a4v/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9772235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsf_bmi5/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9773139Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9774025Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9775192Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:48:19.9775480Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T10:48:19.9776646Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T10:48:19.9776878Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T10:48:19.9777118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9777361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9778281Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9779186Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9780110Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9781003Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9781886Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9782764Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9783629Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9784505Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9785427Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9786295Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T10:48:19.9786407Z ok (3.877s)
2022-12-01T10:48:19.9786428Z 
2022-12-01T10:48:19.9786685Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9786803Z Ran 1 test in 3.878s
2022-12-01T10:48:19.9786823Z 
2022-12-01T10:48:19.9786917Z OK
2022-12-01T10:48:19.9786937Z 
2022-12-01T10:48:19.9787062Z Generating XML reports...
2022-12-01T10:48:19.9787512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103304.xml
2022-12-01T10:48:19.9787884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9788069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9788451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9788646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9788670Z 
2022-12-01T10:48:19.9788762Z Running tests...
2022-12-01T10:48:19.9789089Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9789606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9789871Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9790626Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2022-12-01T10:48:19.9790647Z 
2022-12-01T10:48:19.9790912Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9791026Z Ran 1 test in 1.652s
2022-12-01T10:48:19.9791046Z 
2022-12-01T10:48:19.9791154Z OK (skipped=1)
2022-12-01T10:48:19.9791178Z 
2022-12-01T10:48:19.9791305Z Generating XML reports...
2022-12-01T10:48:19.9791740Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103311.xml
2022-12-01T10:48:19.9792116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9792296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9792681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9792876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9792896Z 
2022-12-01T10:48:19.9793007Z Running tests...
2022-12-01T10:48:19.9793279Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9793596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9793965Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9794168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32810
2022-12-01T10:48:19.9794387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32811
2022-12-01T10:48:19.9794765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9794943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9795326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9795520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9795884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9796065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9796424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9796618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9796864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9797110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9797513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9797915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9798148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9798382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9798714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jm0s0zm
2022-12-01T10:48:19.9798980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jm0s0zm/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9799234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptodnyjzz
2022-12-01T10:48:19.9799505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptodnyjzz/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9800304Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T10:48:19.9800638Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T10:48:19.9801429Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T10:48:19.9801758Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T10:48:19.9801863Z ok (6.397s)
2022-12-01T10:48:19.9801884Z 
2022-12-01T10:48:19.9802149Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9802263Z Ran 1 test in 6.397s
2022-12-01T10:48:19.9802283Z 
2022-12-01T10:48:19.9802377Z OK
2022-12-01T10:48:19.9802396Z 
2022-12-01T10:48:19.9802503Z Generating XML reports...
2022-12-01T10:48:19.9802957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103315.xml
2022-12-01T10:48:19.9803395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9803576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9803958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9804154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9804174Z 
2022-12-01T10:48:19.9804286Z Running tests...
2022-12-01T10:48:19.9804552Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9804850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9805128Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9805878Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.733s)
2022-12-01T10:48:19.9805903Z 
2022-12-01T10:48:19.9806170Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9806286Z Ran 1 test in 1.733s
2022-12-01T10:48:19.9806306Z 
2022-12-01T10:48:19.9806415Z OK (skipped=1)
2022-12-01T10:48:19.9806434Z 
2022-12-01T10:48:19.9806562Z Generating XML reports...
2022-12-01T10:48:19.9807017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103323.xml
2022-12-01T10:48:19.9807388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9807565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9808018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9808224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9808244Z 
2022-12-01T10:48:19.9808356Z Running tests...
2022-12-01T10:48:19.9808628Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9808946Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9809223Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9809966Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.760s)
2022-12-01T10:48:19.9809993Z 
2022-12-01T10:48:19.9810264Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9810382Z Ran 1 test in 1.760s
2022-12-01T10:48:19.9810402Z 
2022-12-01T10:48:19.9810511Z OK (skipped=1)
2022-12-01T10:48:19.9810532Z 
2022-12-01T10:48:19.9810639Z Generating XML reports...
2022-12-01T10:48:19.9811088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103327.xml
2022-12-01T10:48:19.9811466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9811643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9812023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9812216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9812284Z 
2022-12-01T10:48:19.9812400Z Running tests...
2022-12-01T10:48:19.9812673Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9812972Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9813267Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9813489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33517
2022-12-01T10:48:19.9813709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33518
2022-12-01T10:48:19.9814082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9814261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9814645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9814840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9815212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9815369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9815747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9815938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9816186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9816433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9816834Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9817282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9817525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9817756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9817980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:19.9818222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:19.9818629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9819032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:19.9819293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt38ab8b1
2022-12-01T10:48:19.9819576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt38ab8b1/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9819830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyuuf7xn
2022-12-01T10:48:19.9820099Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyuuf7xn/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9820335Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9820556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9820789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9821018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9821123Z ok (6.603s)
2022-12-01T10:48:19.9821143Z 
2022-12-01T10:48:19.9821412Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9821579Z Ran 1 test in 6.603s
2022-12-01T10:48:19.9821599Z 
2022-12-01T10:48:19.9821699Z OK
2022-12-01T10:48:19.9821719Z 
2022-12-01T10:48:19.9821847Z Generating XML reports...
2022-12-01T10:48:19.9822284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103331.xml
2022-12-01T10:48:19.9822659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9822838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9823217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9823410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9823430Z 
2022-12-01T10:48:19.9823541Z Running tests...
2022-12-01T10:48:19.9823804Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9824127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9824388Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9824613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33851
2022-12-01T10:48:19.9824834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33852
2022-12-01T10:48:19.9825212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9825391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9825773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9825968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9826382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9826567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9826933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9827124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9827438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9827686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9828088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9828485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9828729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9829009Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:48:19.9829240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9829727Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:48:19.9829993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsx4ycp4e
2022-12-01T10:48:19.9830269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsx4ycp4e/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9830528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpekhemi_b
2022-12-01T10:48:19.9830804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpekhemi_b/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9831136Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9831377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9831606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9831822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9832103Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:48:19.9832380Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:48:19.9832658Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:48:19.9832929Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T10:48:19.9833172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9833408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9833640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9833871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9834125Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:48:19.9834406Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T10:48:19.9834681Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T10:48:19.9834950Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T10:48:19.9835249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9835494Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9835728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9835959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9836065Z ok (6.887s)
2022-12-01T10:48:19.9836085Z 
2022-12-01T10:48:19.9836350Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9836468Z Ran 1 test in 6.887s
2022-12-01T10:48:19.9836488Z 
2022-12-01T10:48:19.9836584Z OK
2022-12-01T10:48:19.9836603Z 
2022-12-01T10:48:19.9836729Z Generating XML reports...
2022-12-01T10:48:19.9837185Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103340.xml
2022-12-01T10:48:19.9837575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9837758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9838141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9838337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9838357Z 
2022-12-01T10:48:19.9838451Z Running tests...
2022-12-01T10:48:19.9838720Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9839037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9839309Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9840059Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.789s)
2022-12-01T10:48:19.9840127Z 
2022-12-01T10:48:19.9840406Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9840524Z Ran 1 test in 1.789s
2022-12-01T10:48:19.9840544Z 
2022-12-01T10:48:19.9840655Z OK (skipped=1)
2022-12-01T10:48:19.9840675Z 
2022-12-01T10:48:19.9840802Z Generating XML reports...
2022-12-01T10:48:19.9841243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103349.xml
2022-12-01T10:48:19.9841622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9841801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9842182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9842384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9842404Z 
2022-12-01T10:48:19.9842515Z Running tests...
2022-12-01T10:48:19.9842782Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9843093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9843373Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9843578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34255
2022-12-01T10:48:19.9843800Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34256
2022-12-01T10:48:19.9844173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9844358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9844784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9844989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9845358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9845533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9845890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9846084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9846335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9846583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9846997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9847400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9847633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9847864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9848415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:48:19.9848966Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T10:48:19.9849284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt2hpgho
2022-12-01T10:48:19.9849540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt2hpgho/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9849795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56ek497c
2022-12-01T10:48:19.9850062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56ek497c/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9850299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9850537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9850819Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T10:48:19.9851099Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T10:48:19.9851398Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T10:48:19.9851696Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T10:48:19.9852006Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T10:48:19.9852336Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T10:48:19.9852704Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T10:48:19.9853035Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T10:48:19.9853273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9853508Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9853612Z ok (6.314s)
2022-12-01T10:48:19.9853632Z 
2022-12-01T10:48:19.9853911Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9854027Z Ran 1 test in 6.314s
2022-12-01T10:48:19.9854047Z 
2022-12-01T10:48:19.9854121Z OK
2022-12-01T10:48:19.9854161Z 
2022-12-01T10:48:19.9854267Z Generating XML reports...
2022-12-01T10:48:19.9854724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103353.xml
2022-12-01T10:48:19.9855110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9855288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9855671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9855867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9855888Z 
2022-12-01T10:48:19.9855997Z Running tests...
2022-12-01T10:48:19.9856261Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9856559Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9856874Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9857150Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34655
2022-12-01T10:48:19.9857372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34656
2022-12-01T10:48:19.9857749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9857929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9858313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9858504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9858875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9859030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9859404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9859603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9859852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9860099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9860498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9860897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9861129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9861358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9861601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdllukwp_
2022-12-01T10:48:19.9861934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdllukwp_/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9862202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipzipzzt
2022-12-01T10:48:19.9862474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipzipzzt/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9862709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9862944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9863176Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9863403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9863488Z ok (6.301s)
2022-12-01T10:48:19.9863507Z 
2022-12-01T10:48:19.9863782Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9863900Z Ran 1 test in 6.301s
2022-12-01T10:48:19.9863923Z 
2022-12-01T10:48:19.9864017Z OK
2022-12-01T10:48:19.9864036Z 
2022-12-01T10:48:19.9864161Z Generating XML reports...
2022-12-01T10:48:19.9864615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103401.xml
2022-12-01T10:48:19.9864987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9865165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9865524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9865720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9865740Z 
2022-12-01T10:48:19.9865850Z Running tests...
2022-12-01T10:48:19.9866115Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9866492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9866808Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9867030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35037
2022-12-01T10:48:19.9867248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35038
2022-12-01T10:48:19.9867618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9867776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9868160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9868352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9868724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9868901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9869275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9869702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9869955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9870202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9870597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9870993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9871304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9871544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9871804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sygjpv2
2022-12-01T10:48:19.9872075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sygjpv2/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9872328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xgljxmv
2022-12-01T10:48:19.9872595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xgljxmv/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9872812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9873048Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9873284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9873516Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9873619Z ok (6.366s)
2022-12-01T10:48:19.9873639Z 
2022-12-01T10:48:19.9873916Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9874032Z Ran 1 test in 6.366s
2022-12-01T10:48:19.9874052Z 
2022-12-01T10:48:19.9874147Z OK
2022-12-01T10:48:19.9874166Z 
2022-12-01T10:48:19.9874290Z Generating XML reports...
2022-12-01T10:48:19.9874728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103410.xml
2022-12-01T10:48:19.9875105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9875283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9875665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9875943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9875964Z 
2022-12-01T10:48:19.9876075Z Running tests...
2022-12-01T10:48:19.9876344Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9876658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9877014Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9877236Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35435
2022-12-01T10:48:19.9877454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35436
2022-12-01T10:48:19.9877827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9878054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9878441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9878634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9878997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9879171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9879525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9879718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9879963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9880262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9880676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9881076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9881307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9881537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9881800Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1depz8u
2022-12-01T10:48:19.9882053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1depz8u/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9882308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2svwtcz
2022-12-01T10:48:19.9882586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2svwtcz/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9882822Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9883060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9883294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9883528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9883629Z ok (6.466s)
2022-12-01T10:48:19.9883650Z 
2022-12-01T10:48:19.9883898Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9884014Z Ran 1 test in 6.467s
2022-12-01T10:48:19.9884033Z 
2022-12-01T10:48:19.9884127Z OK
2022-12-01T10:48:19.9884146Z 
2022-12-01T10:48:19.9884271Z Generating XML reports...
2022-12-01T10:48:19.9884791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103418.xml
2022-12-01T10:48:19.9885170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9885349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9885732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9885925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9885946Z 
2022-12-01T10:48:19.9886036Z Running tests...
2022-12-01T10:48:19.9886300Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9886619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9886991Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9887219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35834
2022-12-01T10:48:19.9887437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35835
2022-12-01T10:48:19.9887811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9887990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9888372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9888546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9888923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9889103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9889530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9889729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9889974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9890219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9890621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9891002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9891237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9891466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9891735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8xzd6fp
2022-12-01T10:48:19.9892010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8xzd6fp/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9892265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwm5otit
2022-12-01T10:48:19.9892533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwm5otit/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9892768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9893008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9893222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9893456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9893616Z ok (16.740s)
2022-12-01T10:48:19.9893637Z 
2022-12-01T10:48:19.9893913Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9894030Z Ran 1 test in 16.740s
2022-12-01T10:48:19.9894050Z 
2022-12-01T10:48:19.9894144Z OK
2022-12-01T10:48:19.9894164Z 
2022-12-01T10:48:19.9894290Z Generating XML reports...
2022-12-01T10:48:19.9894742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103427.xml
2022-12-01T10:48:19.9895096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9895273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9895656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9895850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9895874Z 
2022-12-01T10:48:19.9895986Z Running tests...
2022-12-01T10:48:19.9896254Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9896574Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9896946Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9897165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36271
2022-12-01T10:48:19.9897365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36272
2022-12-01T10:48:19.9897740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9897918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9898300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9898547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9898925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9899100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9899476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9899667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9899895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9900140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9900544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9900951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9901183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9901414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9901673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgd4fv5uv
2022-12-01T10:48:19.9901948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgd4fv5uv/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9902202Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkdxlsbu
2022-12-01T10:48:19.9902455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkdxlsbu/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9902697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9902993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9903228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9903455Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9903558Z ok (6.664s)
2022-12-01T10:48:19.9903579Z 
2022-12-01T10:48:19.9903849Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9903964Z Ran 1 test in 6.664s
2022-12-01T10:48:19.9903984Z 
2022-12-01T10:48:19.9904059Z OK
2022-12-01T10:48:19.9904078Z 
2022-12-01T10:48:19.9904203Z Generating XML reports...
2022-12-01T10:48:19.9904656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103446.xml
2022-12-01T10:48:19.9905028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9905216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9905595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9905787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9905808Z 
2022-12-01T10:48:19.9905918Z Running tests...
2022-12-01T10:48:19.9906183Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9906479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9906849Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9907074Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36608
2022-12-01T10:48:19.9907298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36609
2022-12-01T10:48:19.9907713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9907900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9908273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9908447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9908805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9908998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9909595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9909799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9910058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9910306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9910718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9911119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9911351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9911557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9911820Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb2ii8ag
2022-12-01T10:48:19.9912090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb2ii8ag/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9912432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rs9_enq
2022-12-01T10:48:19.9912703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rs9_enq/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9912943Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9913181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9913414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9913646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9913730Z ok (6.583s)
2022-12-01T10:48:19.9913751Z 
2022-12-01T10:48:19.9914025Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9914141Z Ran 1 test in 6.583s
2022-12-01T10:48:19.9914164Z 
2022-12-01T10:48:19.9914258Z OK
2022-12-01T10:48:19.9914278Z 
2022-12-01T10:48:19.9914407Z Generating XML reports...
2022-12-01T10:48:19.9914862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103455.xml
2022-12-01T10:48:19.9915236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9915416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9915776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9915974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9915994Z 
2022-12-01T10:48:19.9916104Z Running tests...
2022-12-01T10:48:19.9916368Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9916682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9917115Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9917345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37039
2022-12-01T10:48:19.9917561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37040
2022-12-01T10:48:19.9917938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9918096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9918474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9918666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9919032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9919217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9919600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9919794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9920039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9920266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9920669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9921064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9921357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9921592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9921850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_zt59va
2022-12-01T10:48:19.9922120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_zt59va/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9922375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi1cqct47
2022-12-01T10:48:19.9922646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi1cqct47/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9922861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9923098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9923330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9923569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9923672Z ok (6.595s)
2022-12-01T10:48:19.9923693Z 
2022-12-01T10:48:19.9923965Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9924079Z Ran 1 test in 6.595s
2022-12-01T10:48:19.9924099Z 
2022-12-01T10:48:19.9924192Z OK
2022-12-01T10:48:19.9924211Z 
2022-12-01T10:48:19.9924335Z Generating XML reports...
2022-12-01T10:48:19.9924771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103504.xml
2022-12-01T10:48:19.9925148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9925328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9925710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9925967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9925988Z 
2022-12-01T10:48:19.9926106Z Running tests...
2022-12-01T10:48:19.9926376Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9926691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9927044Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9927265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37435
2022-12-01T10:48:19.9927484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37436
2022-12-01T10:48:19.9927862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9928044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9928427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9928624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9928991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9929166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9929523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9929714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9929962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9930207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9930669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9931075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9931307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9931537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9931797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuu4w6f42
2022-12-01T10:48:19.9932066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuu4w6f42/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9932322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb5cx5nev
2022-12-01T10:48:19.9932599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb5cx5nev/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9932819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9933055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9933284Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9933515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9933617Z ok (6.405s)
2022-12-01T10:48:19.9933638Z 
2022-12-01T10:48:19.9933910Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9934026Z Ran 1 test in 6.405s
2022-12-01T10:48:19.9934046Z 
2022-12-01T10:48:19.9934140Z OK
2022-12-01T10:48:19.9934160Z 
2022-12-01T10:48:19.9934266Z Generating XML reports...
2022-12-01T10:48:19.9934721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103513.xml
2022-12-01T10:48:19.9935139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9935324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9935707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9935899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9935920Z 
2022-12-01T10:48:19.9936029Z Running tests...
2022-12-01T10:48:19.9936298Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9936611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9936962Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9937193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37811
2022-12-01T10:48:19.9937411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37812
2022-12-01T10:48:19.9937786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9937962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9938340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9938532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9938897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9939073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9939491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9939679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9939924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9940167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9940567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9940965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9941195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9941418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9941687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmk1dtm_
2022-12-01T10:48:19.9941944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmk1dtm_/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9942197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy25ytb2a
2022-12-01T10:48:19.9942467Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy25ytb2a/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9942699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9942933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9943165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9943398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9943498Z ok (6.449s)
2022-12-01T10:48:19.9943521Z 
2022-12-01T10:48:19.9943817Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9943942Z Ran 1 test in 6.450s
2022-12-01T10:48:19.9943962Z 
2022-12-01T10:48:19.9944056Z OK
2022-12-01T10:48:19.9944075Z 
2022-12-01T10:48:19.9944199Z Generating XML reports...
2022-12-01T10:48:19.9944651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103521.xml
2022-12-01T10:48:19.9945022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9945201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9945586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9945783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9945803Z 
2022-12-01T10:48:19.9945899Z Running tests...
2022-12-01T10:48:19.9946167Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9946483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9946853Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9947071Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38141
2022-12-01T10:48:19.9947290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38142
2022-12-01T10:48:19.9947663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9947842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9948205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9948495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9948866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9949039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9949632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9949834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9950079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9950324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9950733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9951123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9951352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9951581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9951839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9tlrtok
2022-12-01T10:48:19.9952112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9tlrtok/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9952364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnn7hx6r
2022-12-01T10:48:19.9952638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnn7hx6r/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9952874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9953187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9953413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9953645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9953749Z ok (6.545s)
2022-12-01T10:48:19.9953770Z 
2022-12-01T10:48:19.9954041Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9954154Z Ran 1 test in 6.545s
2022-12-01T10:48:19.9954174Z 
2022-12-01T10:48:19.9954268Z OK
2022-12-01T10:48:19.9954288Z 
2022-12-01T10:48:19.9954412Z Generating XML reports...
2022-12-01T10:48:19.9954869Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103530.xml
2022-12-01T10:48:19.9955223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9955404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9955792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9955983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9956002Z 
2022-12-01T10:48:19.9956111Z Running tests...
2022-12-01T10:48:19.9956377Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9956695Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9957008Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9957228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38514
2022-12-01T10:48:19.9957428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38515
2022-12-01T10:48:19.9957886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9958063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9958443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9958639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9958999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9959172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9959543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9959718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9959963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9960216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9960620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9961019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9961252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9961482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9961738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskudg2sc
2022-12-01T10:48:19.9962012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskudg2sc/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9962252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbft1ble_
2022-12-01T10:48:19.9962567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbft1ble_/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9962810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9963044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9963277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9963510Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9963613Z ok (6.340s)
2022-12-01T10:48:19.9963633Z 
2022-12-01T10:48:19.9963901Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9963996Z Ran 1 test in 6.340s
2022-12-01T10:48:19.9964035Z 
2022-12-01T10:48:19.9964109Z OK
2022-12-01T10:48:19.9964128Z 
2022-12-01T10:48:19.9964259Z Generating XML reports...
2022-12-01T10:48:19.9964717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103539.xml
2022-12-01T10:48:19.9965092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9965269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9965651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9965845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9965864Z 
2022-12-01T10:48:19.9965973Z Running tests...
2022-12-01T10:48:19.9966220Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9966539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9966856Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9967145Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38855
2022-12-01T10:48:19.9967363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38856
2022-12-01T10:48:19.9967736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9967913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9968291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9968484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9968831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9969008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9969387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9969580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9969829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9970075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9970477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9970878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9971093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9971325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9971626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75mbq5iz
2022-12-01T10:48:19.9971904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75mbq5iz/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9972158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwz6i6g8l
2022-12-01T10:48:19.9972425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwz6i6g8l/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9972660Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9972898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9973130Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9973342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9973451Z ok (6.443s)
2022-12-01T10:48:19.9973471Z 
2022-12-01T10:48:19.9973745Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9973864Z Ran 1 test in 6.443s
2022-12-01T10:48:19.9973883Z 
2022-12-01T10:48:19.9973976Z OK
2022-12-01T10:48:19.9973996Z 
2022-12-01T10:48:19.9974119Z Generating XML reports...
2022-12-01T10:48:19.9974573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103548.xml
2022-12-01T10:48:19.9974946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9975104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9975484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9975677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9975742Z 
2022-12-01T10:48:19.9975855Z Running tests...
2022-12-01T10:48:19.9976122Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9976437Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9976703Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9977453Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.604s)
2022-12-01T10:48:19.9977475Z 
2022-12-01T10:48:19.9977742Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9977853Z Ran 1 test in 1.604s
2022-12-01T10:48:19.9977873Z 
2022-12-01T10:48:19.9977966Z OK (skipped=1)
2022-12-01T10:48:19.9977986Z 
2022-12-01T10:48:19.9978160Z Generating XML reports...
2022-12-01T10:48:19.9978617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103556.xml
2022-12-01T10:48:19.9978989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9979167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9979548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9979744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9979765Z 
2022-12-01T10:48:19.9979871Z Running tests...
2022-12-01T10:48:19.9980119Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9980439Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9980746Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9980975Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39384
2022-12-01T10:48:19.9981191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39385
2022-12-01T10:48:19.9981565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9981742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9982126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9982316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9982662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9982835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9983212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9983401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9983648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9983894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9984296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9984696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9984928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9985196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9985458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwuql2hxj
2022-12-01T10:48:19.9985731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwuql2hxj/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9985983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7vjm9q4g
2022-12-01T10:48:19.9986251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7vjm9q4g/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9986355Z ok (6.390s)
2022-12-01T10:48:19.9986375Z 
2022-12-01T10:48:19.9986643Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9986757Z Ran 1 test in 6.391s
2022-12-01T10:48:19.9986777Z 
2022-12-01T10:48:19.9986852Z OK
2022-12-01T10:48:19.9986871Z 
2022-12-01T10:48:19.9986996Z Generating XML reports...
2022-12-01T10:48:19.9987450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103600.xml
2022-12-01T10:48:19.9987826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9988002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9988383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9988575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9988594Z 
2022-12-01T10:48:19.9988705Z Running tests...
2022-12-01T10:48:19.9988972Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9989266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:19.9989787Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:19.9990022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39767
2022-12-01T10:48:19.9990322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39768
2022-12-01T10:48:19.9990714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9990892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9991277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9991469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9991814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9991992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9992365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9992563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9992809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:19.9993053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:19.9993455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9993854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:19.9994085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:19.9994297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:19.9994554Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqp_bwu_7
2022-12-01T10:48:19.9994897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqp_bwu_7/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9995154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpja7cllbd
2022-12-01T10:48:19.9995422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpja7cllbd/_remote_module_non_scriptable.py
2022-12-01T10:48:19.9995658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9995896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:19.9996300Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpja7cllbd'>
2022-12-01T10:48:19.9996468Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:48:19.9996851Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqp_bwu_7'>
2022-12-01T10:48:19.9997022Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T10:48:19.9997125Z ok (6.080s)
2022-12-01T10:48:19.9997145Z 
2022-12-01T10:48:19.9997414Z ----------------------------------------------------------------------
2022-12-01T10:48:19.9997528Z Ran 1 test in 6.081s
2022-12-01T10:48:19.9997548Z 
2022-12-01T10:48:19.9997642Z OK
2022-12-01T10:48:19.9997661Z 
2022-12-01T10:48:19.9997788Z Generating XML reports...
2022-12-01T10:48:19.9998239Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103609.xml
2022-12-01T10:48:19.9998593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:19.9998771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:19.9999156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:19.9999400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:19.9999422Z 
2022-12-01T10:48:19.9999536Z Running tests...
2022-12-01T10:48:19.9999799Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0000114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0000384Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0000586Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40098
2022-12-01T10:48:20.0000808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40099
2022-12-01T10:48:20.0001182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0001359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0001751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0001945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0002317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0002493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0002867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0003041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0003285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0003529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0003928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0004387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0004621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0004852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0005108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ca308p3
2022-12-01T10:48:20.0005365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_76j_hd
2022-12-01T10:48:20.0005618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ca308p3/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0005882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_76j_hd/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0006123Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0006363Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0006468Z ok (4.149s)
2022-12-01T10:48:20.0006488Z 
2022-12-01T10:48:20.0006760Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0006876Z Ran 1 test in 4.150s
2022-12-01T10:48:20.0006896Z 
2022-12-01T10:48:20.0006990Z OK
2022-12-01T10:48:20.0007009Z 
2022-12-01T10:48:20.0007114Z Generating XML reports...
2022-12-01T10:48:20.0007569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103617.xml
2022-12-01T10:48:20.0007942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0008121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0008501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0008741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0008762Z 
2022-12-01T10:48:20.0008875Z Running tests...
2022-12-01T10:48:20.0009141Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0009453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0009701Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0009920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40393
2022-12-01T10:48:20.0010140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40394
2022-12-01T10:48:20.0010511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0010693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0011077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0011272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0011642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0011800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0012176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0012365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0012618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0012860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0013321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0013722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0013956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0014183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0014425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzs8zbmt0
2022-12-01T10:48:20.0014694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzs8zbmt0/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0014948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00m6evq6
2022-12-01T10:48:20.0015215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00m6evq6/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0015458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0015692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0015796Z ok (6.303s)
2022-12-01T10:48:20.0015816Z 
2022-12-01T10:48:20.0016084Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0016180Z Ran 1 test in 6.303s
2022-12-01T10:48:20.0016219Z 
2022-12-01T10:48:20.0016294Z OK
2022-12-01T10:48:20.0016312Z 
2022-12-01T10:48:20.0016437Z Generating XML reports...
2022-12-01T10:48:20.0016891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103623.xml
2022-12-01T10:48:20.0017268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0017448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0017877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0018077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0018097Z 
2022-12-01T10:48:20.0018206Z Running tests...
2022-12-01T10:48:20.0018453Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0018769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0019057Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0019278Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40789
2022-12-01T10:48:20.0019494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40790
2022-12-01T10:48:20.0019863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0020050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0020432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0020607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0020975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0021150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0021529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0021717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0021963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0022261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0022663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0023061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0023275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0023504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0023745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0023988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0024388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0024784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0025024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:20.0025260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:20.0025649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0026023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0026286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92str2cv
2022-12-01T10:48:20.0026559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92str2cv/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0026862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxj_3cd9
2022-12-01T10:48:20.0027134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxj_3cd9/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0027237Z ok (5.637s)
2022-12-01T10:48:20.0027258Z 
2022-12-01T10:48:20.0027527Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0027640Z Ran 1 test in 5.638s
2022-12-01T10:48:20.0027660Z 
2022-12-01T10:48:20.0027754Z OK
2022-12-01T10:48:20.0027774Z 
2022-12-01T10:48:20.0027878Z Generating XML reports...
2022-12-01T10:48:20.0028333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103632.xml
2022-12-01T10:48:20.0028708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0028886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0029276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0029839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0029862Z 
2022-12-01T10:48:20.0029980Z Running tests...
2022-12-01T10:48:20.0030257Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0030557Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0030839Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0031056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41081
2022-12-01T10:48:20.0031276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41082
2022-12-01T10:48:20.0031646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0031916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0032299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0032489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0032858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0033015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0033397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0033585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0033831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0034076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0034478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0034876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0035110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0035338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0035562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0035803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0036199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0036655Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0036907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:20.0037140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:20.0037533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0037924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0038181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnncpqyva
2022-12-01T10:48:20.0038438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnncpqyva/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0038698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8vq7rak
2022-12-01T10:48:20.0038967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8vq7rak/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0039067Z ok (15.714s)
2022-12-01T10:48:20.0039087Z 
2022-12-01T10:48:20.0039355Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0039466Z Ran 1 test in 15.715s
2022-12-01T10:48:20.0039486Z 
2022-12-01T10:48:20.0039578Z OK
2022-12-01T10:48:20.0039597Z 
2022-12-01T10:48:20.0039723Z Generating XML reports...
2022-12-01T10:48:20.0040157Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103640.xml
2022-12-01T10:48:20.0040531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0040709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0041149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0041337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0041356Z 
2022-12-01T10:48:20.0041462Z Running tests...
2022-12-01T10:48:20.0041724Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0042036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0042342Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0042543Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41664
2022-12-01T10:48:20.0042760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41665
2022-12-01T10:48:20.0043130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0043311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0043690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0043882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0044245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0044418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0044788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0044960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0045206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0045454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0045904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0046310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0046540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0046771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0047026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6faspqn1
2022-12-01T10:48:20.0047297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6faspqn1/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0047531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7hwoz6xy
2022-12-01T10:48:20.0047807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7hwoz6xy/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0047910Z ok (6.408s)
2022-12-01T10:48:20.0047930Z 
2022-12-01T10:48:20.0048204Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0048318Z Ran 1 test in 6.408s
2022-12-01T10:48:20.0048338Z 
2022-12-01T10:48:20.0048433Z OK
2022-12-01T10:48:20.0048452Z 
2022-12-01T10:48:20.0048576Z Generating XML reports...
2022-12-01T10:48:20.0049026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103657.xml
2022-12-01T10:48:20.0049380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0049557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0049938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0050182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0050205Z 
2022-12-01T10:48:20.0050314Z Running tests...
2022-12-01T10:48:20.0050577Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0050893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0051186Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0051407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41968
2022-12-01T10:48:20.0051608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41969
2022-12-01T10:48:20.0051982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0052156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0052538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0052728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0053090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0053263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0053643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0053815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0054059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0054302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0054700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0055156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0055394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0055621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0055876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8_cfguj
2022-12-01T10:48:20.0056147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8_cfguj/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0056382Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy89h2x8s
2022-12-01T10:48:20.0056647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy89h2x8s/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0056754Z ok (6.407s)
2022-12-01T10:48:20.0056773Z 
2022-12-01T10:48:20.0057045Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0057157Z Ran 1 test in 6.407s
2022-12-01T10:48:20.0057177Z 
2022-12-01T10:48:20.0057269Z OK
2022-12-01T10:48:20.0057288Z 
2022-12-01T10:48:20.0057413Z Generating XML reports...
2022-12-01T10:48:20.0057864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103706.xml
2022-12-01T10:48:20.0058216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0058394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0058773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0058962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0059026Z 
2022-12-01T10:48:20.0059140Z Running tests...
2022-12-01T10:48:20.0059409Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0059725Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0059986Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0060206Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42285
2022-12-01T10:48:20.0060404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42286
2022-12-01T10:48:20.0060777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0060952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0061331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0061525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0061892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0062065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0062447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0062616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0062862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0063104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0063504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0063900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0064177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0064413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0064670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj96_yd3j
2022-12-01T10:48:20.0064939Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj96_yd3j/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0065176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk5nbsp0_
2022-12-01T10:48:20.0065443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk5nbsp0_/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0065546Z ok (6.078s)
2022-12-01T10:48:20.0065565Z 
2022-12-01T10:48:20.0065832Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0065950Z Ran 1 test in 6.078s
2022-12-01T10:48:20.0065970Z 
2022-12-01T10:48:20.0066062Z OK
2022-12-01T10:48:20.0066084Z 
2022-12-01T10:48:20.0066208Z Generating XML reports...
2022-12-01T10:48:20.0066661Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103715.xml
2022-12-01T10:48:20.0067035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0067194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0067573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0067761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0067781Z 
2022-12-01T10:48:20.0067889Z Running tests...
2022-12-01T10:48:20.0068155Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0068524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0068786Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0069006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42673
2022-12-01T10:48:20.0069202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42674
2022-12-01T10:48:20.0069764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0069944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0070331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0070525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0070895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0071076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0071449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0071641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0071868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0072113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0072512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0072909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0073139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0073442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0073709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiosr4_c
2022-12-01T10:48:20.0073983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiosr4_c/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0074222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3nmlvnw
2022-12-01T10:48:20.0074491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3nmlvnw/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0075275Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:20.0076053Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:20.0076292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0076527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0076693Z ok (6.094s)
2022-12-01T10:48:20.0076713Z 
2022-12-01T10:48:20.0076991Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0077104Z Ran 1 test in 6.094s
2022-12-01T10:48:20.0077124Z 
2022-12-01T10:48:20.0077217Z OK
2022-12-01T10:48:20.0077236Z 
2022-12-01T10:48:20.0077361Z Generating XML reports...
2022-12-01T10:48:20.0077815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103723.xml
2022-12-01T10:48:20.0078232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0078392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0078774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0078965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0078988Z 
2022-12-01T10:48:20.0079098Z Running tests...
2022-12-01T10:48:20.0079367Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0079684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0079965Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0080712Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.725s)
2022-12-01T10:48:20.0080733Z 
2022-12-01T10:48:20.0080995Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0081088Z Ran 1 test in 1.725s
2022-12-01T10:48:20.0081130Z 
2022-12-01T10:48:20.0081220Z OK (skipped=1)
2022-12-01T10:48:20.0081243Z 
2022-12-01T10:48:20.0081367Z Generating XML reports...
2022-12-01T10:48:20.0081862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103732.xml
2022-12-01T10:48:20.0082243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0082421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0082803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0082995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0083015Z 
2022-12-01T10:48:20.0083126Z Running tests...
2022-12-01T10:48:20.0083372Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0083687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0083975Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0084722Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s)
2022-12-01T10:48:20.0084742Z 
2022-12-01T10:48:20.0085003Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0085119Z Ran 1 test in 1.649s
2022-12-01T10:48:20.0085138Z 
2022-12-01T10:48:20.0085245Z OK (skipped=1)
2022-12-01T10:48:20.0085263Z 
2022-12-01T10:48:20.0085388Z Generating XML reports...
2022-12-01T10:48:20.0085837Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103736.xml
2022-12-01T10:48:20.0086206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0086420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0086805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0086996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0087016Z 
2022-12-01T10:48:20.0087125Z Running tests...
2022-12-01T10:48:20.0087388Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0087702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0087983Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0088201Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43231
2022-12-01T10:48:20.0088404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43232
2022-12-01T10:48:20.0088779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0088953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0089331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0089522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0089885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0090059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0090431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0090619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0090932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0091185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0091588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0091985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0092215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0092440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0092694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8r1_22r
2022-12-01T10:48:20.0092959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8r1_22r/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0093204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv04hu8qz
2022-12-01T10:48:20.0093471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv04hu8qz/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0093815Z STAGE:2022-12-01 10:37:45 43231:43231 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0094148Z STAGE:2022-12-01 10:37:45 43232:43232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0094386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0094618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0094955Z STAGE:2022-12-01 10:37:45 43231:43231 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0095285Z STAGE:2022-12-01 10:37:45 43232:43232 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0096118Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:20.0096889Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:20.0097431Z STAGE:2022-12-01 10:37:45 43231:43231 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:37:45 43232:43232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0097451Z 
2022-12-01T10:48:20.0097786Z STAGE:2022-12-01 10:37:45 43231:43231 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0098116Z STAGE:2022-12-01 10:37:45 43232:43232 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0098200Z ok (6.582s)
2022-12-01T10:48:20.0098241Z 
2022-12-01T10:48:20.0098488Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0098598Z Ran 1 test in 6.582s
2022-12-01T10:48:20.0098618Z 
2022-12-01T10:48:20.0098710Z OK
2022-12-01T10:48:20.0098729Z 
2022-12-01T10:48:20.0098852Z Generating XML reports...
2022-12-01T10:48:20.0099350Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103739.xml
2022-12-01T10:48:20.0099732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0099908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0100288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0100464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0100502Z 
2022-12-01T10:48:20.0100593Z Running tests...
2022-12-01T10:48:20.0100853Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0101166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0101435Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0101661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43583
2022-12-01T10:48:20.0101880Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43584
2022-12-01T10:48:20.0102252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0102410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0102790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0102981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0103341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0103514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0103947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0104135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0104381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0104626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0105006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0105404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0105633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0105860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0106125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvr0txtvz
2022-12-01T10:48:20.0106400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvr0txtvz/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0106653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hgu7u6i
2022-12-01T10:48:20.0106917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hgu7u6i/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0107020Z ok (5.690s)
2022-12-01T10:48:20.0107040Z 
2022-12-01T10:48:20.0107293Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0107410Z Ran 1 test in 5.690s
2022-12-01T10:48:20.0107429Z 
2022-12-01T10:48:20.0107522Z OK
2022-12-01T10:48:20.0107541Z 
2022-12-01T10:48:20.0107664Z Generating XML reports...
2022-12-01T10:48:20.0108112Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103748.xml
2022-12-01T10:48:20.0108535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0108718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0109099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0109271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0109308Z 
2022-12-01T10:48:20.0109586Z Running tests...
2022-12-01T10:48:20.0109866Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0110184Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0110464Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0111213Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.706s)
2022-12-01T10:48:20.0111237Z 
2022-12-01T10:48:20.0111502Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0111612Z Ran 1 test in 1.706s
2022-12-01T10:48:20.0111633Z 
2022-12-01T10:48:20.0111739Z OK (skipped=1)
2022-12-01T10:48:20.0111758Z 
2022-12-01T10:48:20.0111882Z Generating XML reports...
2022-12-01T10:48:20.0112309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103756.xml
2022-12-01T10:48:20.0112681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0112855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0113326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0113517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0113536Z 
2022-12-01T10:48:20.0113643Z Running tests...
2022-12-01T10:48:20.0113909Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0114221Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0114505Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0114706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44062
2022-12-01T10:48:20.0114925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44063
2022-12-01T10:48:20.0115299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0115480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0115863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0116052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0116421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0116596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0116951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0117142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0117388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0117631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0118099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0118511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0118742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0118969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0119226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxea34voi
2022-12-01T10:48:20.0119478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxea34voi/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0119737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbt3dap7
2022-12-01T10:48:20.0120009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbt3dap7/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0120929Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:48:20.0121044Z   warnings.warn(
2022-12-01T10:48:20.0121947Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T10:48:20.0122059Z   warnings.warn(
2022-12-01T10:48:20.0122349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0122585Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0122688Z ok (6.204s)
2022-12-01T10:48:20.0122708Z 
2022-12-01T10:48:20.0122959Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0123073Z Ran 1 test in 6.204s
2022-12-01T10:48:20.0123093Z 
2022-12-01T10:48:20.0123186Z OK
2022-12-01T10:48:20.0123205Z 
2022-12-01T10:48:20.0123328Z Generating XML reports...
2022-12-01T10:48:20.0123782Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103800.xml
2022-12-01T10:48:20.0124152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0124330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0124712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0124908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0124927Z 
2022-12-01T10:48:20.0125017Z Running tests...
2022-12-01T10:48:20.0125282Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0125601Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0125879Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0126622Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.705s)
2022-12-01T10:48:20.0126642Z 
2022-12-01T10:48:20.0126906Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0127065Z Ran 1 test in 1.705s
2022-12-01T10:48:20.0127086Z 
2022-12-01T10:48:20.0127197Z OK (skipped=1)
2022-12-01T10:48:20.0127217Z 
2022-12-01T10:48:20.0127393Z Generating XML reports...
2022-12-01T10:48:20.0127833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103809.xml
2022-12-01T10:48:20.0128206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0128383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0128767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0128955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0128975Z 
2022-12-01T10:48:20.0129083Z Running tests...
2022-12-01T10:48:20.0129350Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0129665Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0129938Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0130138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44541
2022-12-01T10:48:20.0130358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44542
2022-12-01T10:48:20.0130733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0130908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0131288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0131480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0131910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0132084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0132438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0132628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0132871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0133114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0133513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0133910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0134147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0134376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0134632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h9niygk
2022-12-01T10:48:20.0134887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h9niygk/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0135142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzgrs9j_
2022-12-01T10:48:20.0135408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzgrs9j_/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0135748Z STAGE:2022-12-01 10:38:18 44542:44542 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0136077Z STAGE:2022-12-01 10:38:18 44541:44541 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0136359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0136599Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T10:48:20.0136942Z STAGE:2022-12-01 10:38:18 44542:44542 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0137271Z STAGE:2022-12-01 10:38:18 44541:44541 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0137580Z STAGE:2022-12-01 10:38:18 44541:44541 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0137905Z STAGE:2022-12-01 10:38:19 44541:44541 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0138007Z ok (7.042s)
2022-12-01T10:48:20.0138027Z 
2022-12-01T10:48:20.0138291Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0138402Z Ran 1 test in 7.042s
2022-12-01T10:48:20.0138425Z 
2022-12-01T10:48:20.0138519Z OK
2022-12-01T10:48:20.0138537Z 
2022-12-01T10:48:20.0138665Z Generating XML reports...
2022-12-01T10:48:20.0139119Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103812.xml
2022-12-01T10:48:20.0139472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0139648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0140032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0140226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0140246Z 
2022-12-01T10:48:20.0140354Z Running tests...
2022-12-01T10:48:20.0140618Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0140931Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0141258Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0141480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44971
2022-12-01T10:48:20.0141679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44972
2022-12-01T10:48:20.0142053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0142227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0142605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0142794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0143158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0143339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0143714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0143882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0144130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0144372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0144773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0145169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0145404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0145692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0145956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6r9vxhaj
2022-12-01T10:48:20.0146227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6r9vxhaj/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0146462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n79dpqi
2022-12-01T10:48:20.0146731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n79dpqi/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0146835Z ok (5.656s)
2022-12-01T10:48:20.0146855Z 
2022-12-01T10:48:20.0147129Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0147242Z Ran 1 test in 5.656s
2022-12-01T10:48:20.0147262Z 
2022-12-01T10:48:20.0147354Z OK
2022-12-01T10:48:20.0147374Z 
2022-12-01T10:48:20.0147498Z Generating XML reports...
2022-12-01T10:48:20.0147957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103822.xml
2022-12-01T10:48:20.0148328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0148485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0148865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0149057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0149076Z 
2022-12-01T10:48:20.0149184Z Running tests...
2022-12-01T10:48:20.0149671Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0150003Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0150279Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0150588Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45360
2022-12-01T10:48:20.0150787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45361
2022-12-01T10:48:20.0151161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0151335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0151713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0151906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0152266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0152438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0152816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0152999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0153228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0153473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0153875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0154271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0154505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0154730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0155052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0092q16g
2022-12-01T10:48:20.0155327Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0092q16g/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0155578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6e9nrfxs
2022-12-01T10:48:20.0155826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6e9nrfxs/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0155928Z ok (5.614s)
2022-12-01T10:48:20.0155948Z 
2022-12-01T10:48:20.0156218Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0156332Z Ran 1 test in 5.615s
2022-12-01T10:48:20.0156351Z 
2022-12-01T10:48:20.0156447Z OK
2022-12-01T10:48:20.0156466Z 
2022-12-01T10:48:20.0156589Z Generating XML reports...
2022-12-01T10:48:20.0157043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103830.xml
2022-12-01T10:48:20.0157425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0157585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0157970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0158165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0158184Z 
2022-12-01T10:48:20.0158291Z Running tests...
2022-12-01T10:48:20.0158553Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0158864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0159139Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0159893Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2022-12-01T10:48:20.0159962Z 
2022-12-01T10:48:20.0160233Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0160344Z Ran 1 test in 1.677s
2022-12-01T10:48:20.0160363Z 
2022-12-01T10:48:20.0160453Z OK (skipped=1)
2022-12-01T10:48:20.0160471Z 
2022-12-01T10:48:20.0160594Z Generating XML reports...
2022-12-01T10:48:20.0161046Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103838.xml
2022-12-01T10:48:20.0161419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0161598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0161981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0162174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0162194Z 
2022-12-01T10:48:20.0162303Z Running tests...
2022-12-01T10:48:20.0162548Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0162866Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0163125Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0163871Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-12-01T10:48:20.0163891Z 
2022-12-01T10:48:20.0164158Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0164316Z Ran 1 test in 1.650s
2022-12-01T10:48:20.0164337Z 
2022-12-01T10:48:20.0164449Z OK (skipped=1)
2022-12-01T10:48:20.0164468Z 
2022-12-01T10:48:20.0164593Z Generating XML reports...
2022-12-01T10:48:20.0165049Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103842.xml
2022-12-01T10:48:20.0165420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0165577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0165962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0166153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0166172Z 
2022-12-01T10:48:20.0166280Z Running tests...
2022-12-01T10:48:20.0166546Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0166865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0167154Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0167892Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.724s)
2022-12-01T10:48:20.0167913Z 
2022-12-01T10:48:20.0168174Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0168285Z Ran 1 test in 1.724s
2022-12-01T10:48:20.0168305Z 
2022-12-01T10:48:20.0168394Z OK (skipped=1)
2022-12-01T10:48:20.0168414Z 
2022-12-01T10:48:20.0168538Z Generating XML reports...
2022-12-01T10:48:20.0169048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103846.xml
2022-12-01T10:48:20.0169419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0169591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0169969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0170158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0170177Z 
2022-12-01T10:48:20.0170285Z Running tests...
2022-12-01T10:48:20.0170527Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0170840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0171135Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0171362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46106
2022-12-01T10:48:20.0171577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46107
2022-12-01T10:48:20.0171945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0172121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0172499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0172689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0173037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0173209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0173626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0173822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0174072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0174314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0174716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0175114Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0175342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0175553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0175819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe9p74o7a
2022-12-01T10:48:20.0176090Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe9p74o7a/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0176345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpi1bwja
2022-12-01T10:48:20.0176614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpi1bwja/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0176716Z ok (21.024s)
2022-12-01T10:48:20.0176736Z 
2022-12-01T10:48:20.0177006Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0177121Z Ran 1 test in 21.024s
2022-12-01T10:48:20.0177140Z 
2022-12-01T10:48:20.0177215Z OK
2022-12-01T10:48:20.0177234Z 
2022-12-01T10:48:20.0177356Z Generating XML reports...
2022-12-01T10:48:20.0177803Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103850.xml
2022-12-01T10:48:20.0178287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0178466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0178849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0179040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0179060Z 
2022-12-01T10:48:20.0179169Z Running tests...
2022-12-01T10:48:20.0179433Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0179729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0179990Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0180210Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46640
2022-12-01T10:48:20.0180437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46641
2022-12-01T10:48:20.0180812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0180989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0181367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0181556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0181903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0182076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0182448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0182639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0182945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0183198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0183598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0183995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0184222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0184427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0184672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0184918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0185315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0185709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0185811Z ok (4.096s)
2022-12-01T10:48:20.0185831Z 
2022-12-01T10:48:20.0186095Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0186207Z Ran 1 test in 4.096s
2022-12-01T10:48:20.0186226Z 
2022-12-01T10:48:20.0186319Z OK
2022-12-01T10:48:20.0186338Z 
2022-12-01T10:48:20.0186444Z Generating XML reports...
2022-12-01T10:48:20.0186893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103913.xml
2022-12-01T10:48:20.0187267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0187506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0187888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0188080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0188099Z 
2022-12-01T10:48:20.0188254Z Running tests...
2022-12-01T10:48:20.0188558Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0188858Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0189122Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0189784Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46862
2022-12-01T10:48:20.0190048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46863
2022-12-01T10:48:20.0190515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0190726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0191160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0191389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0191790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0191948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0192359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0192634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0193002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0193303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0193743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0194176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0194445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0194712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0194935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0195212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0195694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0196173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0196311Z ok (4.003s)
2022-12-01T10:48:20.0196332Z 
2022-12-01T10:48:20.0196631Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0196778Z Ran 1 test in 4.003s
2022-12-01T10:48:20.0196799Z 
2022-12-01T10:48:20.0196926Z OK
2022-12-01T10:48:20.0196946Z 
2022-12-01T10:48:20.0197102Z Generating XML reports...
2022-12-01T10:48:20.0197539Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103919.xml
2022-12-01T10:48:20.0197945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0198282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0198710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0198935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0198955Z 
2022-12-01T10:48:20.0199099Z Running tests...
2022-12-01T10:48:20.0199394Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0199747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0200056Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0200787Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.650s)
2022-12-01T10:48:20.0200878Z 
2022-12-01T10:48:20.0201126Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0201341Z Ran 1 test in 1.651s
2022-12-01T10:48:20.0201361Z 
2022-12-01T10:48:20.0201501Z OK (skipped=1)
2022-12-01T10:48:20.0201520Z 
2022-12-01T10:48:20.0201679Z Generating XML reports...
2022-12-01T10:48:20.0202167Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103925.xml
2022-12-01T10:48:20.0202575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0202785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0203211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0203390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0203467Z 
2022-12-01T10:48:20.0203613Z Running tests...
2022-12-01T10:48:20.0203960Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0204311Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0204622Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0205403Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.668s)
2022-12-01T10:48:20.0205424Z 
2022-12-01T10:48:20.0205731Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0205877Z Ran 1 test in 1.668s
2022-12-01T10:48:20.0205901Z 
2022-12-01T10:48:20.0206020Z OK (skipped=1)
2022-12-01T10:48:20.0206039Z 
2022-12-01T10:48:20.0226500Z Generating XML reports...
2022-12-01T10:48:20.0227051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103929.xml
2022-12-01T10:48:20.0227424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0227603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0227993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0228192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0228215Z 
2022-12-01T10:48:20.0228308Z Running tests...
2022-12-01T10:48:20.0228582Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0228899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0229299Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0229834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47259
2022-12-01T10:48:20.0230065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47260
2022-12-01T10:48:20.0230454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0230635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0230997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0231191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0231555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0231737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0232120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0232312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0232560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0232805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0233209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0233590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0233824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0234154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0234269Z ok (3.907s)
2022-12-01T10:48:20.0234289Z 
2022-12-01T10:48:20.0234563Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0234680Z Ran 1 test in 3.907s
2022-12-01T10:48:20.0234700Z 
2022-12-01T10:48:20.0234794Z OK
2022-12-01T10:48:20.0234814Z 
2022-12-01T10:48:20.0234940Z Generating XML reports...
2022-12-01T10:48:20.0235375Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103933.xml
2022-12-01T10:48:20.0235748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0235927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0236306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0236505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0236526Z 
2022-12-01T10:48:20.0236635Z Running tests...
2022-12-01T10:48:20.0236902Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0237217Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0237461Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0237662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47566
2022-12-01T10:48:20.0237882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47567
2022-12-01T10:48:20.0238256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0238435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0238934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0239129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0239498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0239673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0240032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0240224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0240473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0240719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0241122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0241529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0241761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0242102Z STAGE:2022-12-01 10:39:43 47567:47567 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0242430Z STAGE:2022-12-01 10:39:43 47566:47566 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0242642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0242986Z STAGE:2022-12-01 10:39:43 47566:47566 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0243318Z STAGE:2022-12-01 10:39:43 47567:47567 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0243884Z STAGE:2022-12-01 10:39:43 47567:47567 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0244221Z STAGE:2022-12-01 10:39:43 47566:47566 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0244551Z STAGE:2022-12-01 10:39:43 47566:47566 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0244876Z STAGE:2022-12-01 10:39:43 47567:47567 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0244982Z ok (4.032s)
2022-12-01T10:48:20.0245002Z 
2022-12-01T10:48:20.0245273Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0245371Z Ran 1 test in 4.032s
2022-12-01T10:48:20.0245391Z 
2022-12-01T10:48:20.0245485Z OK
2022-12-01T10:48:20.0245505Z 
2022-12-01T10:48:20.0245633Z Generating XML reports...
2022-12-01T10:48:20.0246322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103939.xml
2022-12-01T10:48:20.0246719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0246895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0247274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0247467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0247487Z 
2022-12-01T10:48:20.0247597Z Running tests...
2022-12-01T10:48:20.0247842Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0248157Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0248411Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0248632Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47846
2022-12-01T10:48:20.0248921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47847
2022-12-01T10:48:20.0249297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0249474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0249857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0250031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0250402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0250579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0250956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0251151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0251404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0251652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0252055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0252456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0252671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0252904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0253009Z ok (4.031s)
2022-12-01T10:48:20.0253029Z 
2022-12-01T10:48:20.0253299Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0253418Z Ran 1 test in 4.031s
2022-12-01T10:48:20.0253438Z 
2022-12-01T10:48:20.0253578Z OK
2022-12-01T10:48:20.0253599Z 
2022-12-01T10:48:20.0253732Z Generating XML reports...
2022-12-01T10:48:20.0254187Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103945.xml
2022-12-01T10:48:20.0254541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0254723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0255102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0255293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0255312Z 
2022-12-01T10:48:20.0255423Z Running tests...
2022-12-01T10:48:20.0255689Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0256013Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0256265Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:48:20.0256285Z 
2022-12-01T10:48:20.0256549Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0256644Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0256663Z 
2022-12-01T10:48:20.0256773Z OK (skipped=1)
2022-12-01T10:48:20.0256792Z 
2022-12-01T10:48:20.0256917Z Generating XML reports...
2022-12-01T10:48:20.0257370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103952.xml
2022-12-01T10:48:20.0257741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0257920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0258361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0258555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0258574Z 
2022-12-01T10:48:20.0258684Z Running tests...
2022-12-01T10:48:20.0258926Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0259239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0259497Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0259717Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48243
2022-12-01T10:48:20.0259937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48244
2022-12-01T10:48:20.0260308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0260490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0260869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0261041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0261409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0261585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0261966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0262157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0262404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0262651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0263105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0263518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0263730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0263960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0264204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0264444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0264839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0265238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0265569Z STAGE:2022-12-01 10:39:58 48244:48244 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0265892Z STAGE:2022-12-01 10:39:58 48243:48243 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0266232Z STAGE:2022-12-01 10:39:58 48244:48244 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0266543Z STAGE:2022-12-01 10:39:58 48243:48243 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0266864Z STAGE:2022-12-01 10:39:58 48244:48244 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0267185Z STAGE:2022-12-01 10:39:58 48243:48243 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0267510Z STAGE:2022-12-01 10:39:58 48243:48243 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0267888Z STAGE:2022-12-01 10:39:58 48244:48244 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0267991Z ok (4.309s)
2022-12-01T10:48:20.0268011Z 
2022-12-01T10:48:20.0268274Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0268390Z Ran 1 test in 4.310s
2022-12-01T10:48:20.0268409Z 
2022-12-01T10:48:20.0268484Z OK
2022-12-01T10:48:20.0268524Z 
2022-12-01T10:48:20.0268631Z Generating XML reports...
2022-12-01T10:48:20.0269088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103954.xml
2022-12-01T10:48:20.0269718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0269907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0270299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0270497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0270517Z 
2022-12-01T10:48:20.0270629Z Running tests...
2022-12-01T10:48:20.0270889Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0271187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0271441Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0271663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48546
2022-12-01T10:48:20.0271883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48547
2022-12-01T10:48:20.0272254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0272429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0272876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0273068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0273422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0273585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0273951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0274130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0274363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0274594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0274989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0275374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0275591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0275800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0275948Z skip: Skipped due to small world size. (3.829s)
2022-12-01T10:48:20.0275968Z 
2022-12-01T10:48:20.0276223Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0276323Z Ran 1 test in 3.829s
2022-12-01T10:48:20.0276343Z 
2022-12-01T10:48:20.0276439Z OK (skipped=1)
2022-12-01T10:48:20.0276458Z 
2022-12-01T10:48:20.0276571Z Generating XML reports...
2022-12-01T10:48:20.0277010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104000.xml
2022-12-01T10:48:20.0277467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0277631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0277993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0278215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0278236Z 
2022-12-01T10:48:20.0278334Z Running tests...
2022-12-01T10:48:20.0278584Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0278882Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0279123Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0279330Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48817
2022-12-01T10:48:20.0279544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48818
2022-12-01T10:48:20.0279902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0280070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0280451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0280642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0281011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0281188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0281557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0281788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0282029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0282256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0282652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0283040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0283259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0283475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0283566Z ok (4.022s)
2022-12-01T10:48:20.0283587Z 
2022-12-01T10:48:20.0283856Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0283974Z Ran 1 test in 4.022s
2022-12-01T10:48:20.0283994Z 
2022-12-01T10:48:20.0284068Z OK
2022-12-01T10:48:20.0284107Z 
2022-12-01T10:48:20.0284214Z Generating XML reports...
2022-12-01T10:48:20.0284668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104007.xml
2022-12-01T10:48:20.0285037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0285213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0285595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0285788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0285808Z 
2022-12-01T10:48:20.0285918Z Running tests...
2022-12-01T10:48:20.0286241Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0286540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0286812Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0287571Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.683s)
2022-12-01T10:48:20.0287592Z 
2022-12-01T10:48:20.0287854Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0287968Z Ran 1 test in 1.683s
2022-12-01T10:48:20.0287987Z 
2022-12-01T10:48:20.0288098Z OK (skipped=1)
2022-12-01T10:48:20.0288117Z 
2022-12-01T10:48:20.0288244Z Generating XML reports...
2022-12-01T10:48:20.0288700Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104013.xml
2022-12-01T10:48:20.0289078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0289256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0289617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0289808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0289828Z 
2022-12-01T10:48:20.0289937Z Running tests...
2022-12-01T10:48:20.0290198Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0290512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0290760Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0291030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49304
2022-12-01T10:48:20.0291258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49305
2022-12-01T10:48:20.0291614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0291791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0292172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0292365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0292731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0292908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0293284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0293482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0293709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0293954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0294358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0294758Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0294989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0295216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0295510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0295757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0296157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0296552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0296638Z ok (4.057s)
2022-12-01T10:48:20.0296657Z 
2022-12-01T10:48:20.0296921Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0297037Z Ran 1 test in 4.057s
2022-12-01T10:48:20.0297056Z 
2022-12-01T10:48:20.0297150Z OK
2022-12-01T10:48:20.0297169Z 
2022-12-01T10:48:20.0297293Z Generating XML reports...
2022-12-01T10:48:20.0297743Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104017.xml
2022-12-01T10:48:20.0298121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0298298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0298662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0298854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0298873Z 
2022-12-01T10:48:20.0298973Z Running tests...
2022-12-01T10:48:20.0299233Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0299545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0299791Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0300011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49536
2022-12-01T10:48:20.0300268Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49537
2022-12-01T10:48:20.0300639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0300798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0301164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0301342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0301699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0301860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0302227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0302407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0302644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0302869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0303257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0303643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0303861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0304075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0304166Z ok (3.946s)
2022-12-01T10:48:20.0304186Z 
2022-12-01T10:48:20.0304438Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0304589Z Ran 1 test in 3.946s
2022-12-01T10:48:20.0304612Z 
2022-12-01T10:48:20.0304693Z OK
2022-12-01T10:48:20.0304713Z 
2022-12-01T10:48:20.0304819Z Generating XML reports...
2022-12-01T10:48:20.0305262Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104023.xml
2022-12-01T10:48:20.0305621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0305783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0306151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0306331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0306351Z 
2022-12-01T10:48:20.0306448Z Running tests...
2022-12-01T10:48:20.0306700Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0307005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0307239Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0307449Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49792
2022-12-01T10:48:20.0307655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49793
2022-12-01T10:48:20.0308012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0308174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0308539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0308718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0309123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0309288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0309921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0310101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0310335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0310567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0310957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0311344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0311570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0311785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0311871Z ok (4.242s)
2022-12-01T10:48:20.0311891Z 
2022-12-01T10:48:20.0312145Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0312246Z Ran 1 test in 4.242s
2022-12-01T10:48:20.0312265Z 
2022-12-01T10:48:20.0312348Z OK
2022-12-01T10:48:20.0312367Z 
2022-12-01T10:48:20.0312480Z Generating XML reports...
2022-12-01T10:48:20.0312919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104029.xml
2022-12-01T10:48:20.0313278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0313440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0313894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0314076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0314096Z 
2022-12-01T10:48:20.0314192Z Running tests...
2022-12-01T10:48:20.0314443Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0314747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0315000Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0315207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50117
2022-12-01T10:48:20.0315413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50118
2022-12-01T10:48:20.0315768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0315934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0316305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0316484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0316840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0317001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0317365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0317542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0317773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0317998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0318451Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0318849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0319065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0319278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0319508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0319733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0320114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0320499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0320584Z ok (3.968s)
2022-12-01T10:48:20.0320604Z 
2022-12-01T10:48:20.0320857Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0320957Z Ran 1 test in 3.968s
2022-12-01T10:48:20.0320977Z 
2022-12-01T10:48:20.0321057Z OK
2022-12-01T10:48:20.0321076Z 
2022-12-01T10:48:20.0321189Z Generating XML reports...
2022-12-01T10:48:20.0321627Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104036.xml
2022-12-01T10:48:20.0321987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0322151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0322521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0322753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0322773Z 
2022-12-01T10:48:20.0322870Z Running tests...
2022-12-01T10:48:20.0323125Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0323424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0323673Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0323881Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50405
2022-12-01T10:48:20.0324087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50406
2022-12-01T10:48:20.0324444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0324603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0324979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0325160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0325516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0325679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0326044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0326221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0326453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0326687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0327117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0327512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0327729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0327944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0328171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0328398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0328779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0329157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0329253Z ok (3.979s)
2022-12-01T10:48:20.0329275Z 
2022-12-01T10:48:20.0329526Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0329626Z Ran 1 test in 3.980s
2022-12-01T10:48:20.0329646Z 
2022-12-01T10:48:20.0329727Z OK
2022-12-01T10:48:20.0329747Z 
2022-12-01T10:48:20.0329859Z Generating XML reports...
2022-12-01T10:48:20.0330298Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104042.xml
2022-12-01T10:48:20.0330656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0330821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0331190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0331364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0331434Z 
2022-12-01T10:48:20.0331533Z Running tests...
2022-12-01T10:48:20.0331789Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0332091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0332344Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0332551Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50670
2022-12-01T10:48:20.0332756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50671
2022-12-01T10:48:20.0333117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0333282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0333644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0333830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0334186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0334348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0334711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0334888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0335120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0335351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0335735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0336168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0336396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0336611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0336854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps64hl74x
2022-12-01T10:48:20.0337111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps64hl74x/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0337352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfujvddg5
2022-12-01T10:48:20.0337611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfujvddg5/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0337708Z ok (6.076s)
2022-12-01T10:48:20.0337729Z 
2022-12-01T10:48:20.0337986Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0338104Z Ran 1 test in 6.076s
2022-12-01T10:48:20.0338124Z 
2022-12-01T10:48:20.0338221Z OK
2022-12-01T10:48:20.0338239Z 
2022-12-01T10:48:20.0338364Z Generating XML reports...
2022-12-01T10:48:20.0338816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104048.xml
2022-12-01T10:48:20.0339188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0339366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0339751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0339925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0339964Z 
2022-12-01T10:48:20.0340055Z Running tests...
2022-12-01T10:48:20.0340386Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0340706Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0340946Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0341167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51005
2022-12-01T10:48:20.0341383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51006
2022-12-01T10:48:20.0341753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0341930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0342294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0342491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0342868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0343046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0343424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0343616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0343863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0344109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0344490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0344891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0345171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0345409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0345515Z ok (4.127s)
2022-12-01T10:48:20.0345535Z 
2022-12-01T10:48:20.0345804Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0345920Z Ran 1 test in 4.127s
2022-12-01T10:48:20.0345939Z 
2022-12-01T10:48:20.0346034Z OK
2022-12-01T10:48:20.0346053Z 
2022-12-01T10:48:20.0346179Z Generating XML reports...
2022-12-01T10:48:20.0346612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104057.xml
2022-12-01T10:48:20.0346984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0347163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0347551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0347746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0347766Z 
2022-12-01T10:48:20.0347876Z Running tests...
2022-12-01T10:48:20.0348140Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0348458Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0348679Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0348901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51305
2022-12-01T10:48:20.0349122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51306
2022-12-01T10:48:20.0349720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0350001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0350400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0350591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0350957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0351135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0351496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0351689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0351937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0352184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0352596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0352997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0353229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0353458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0353563Z ok (3.972s)
2022-12-01T10:48:20.0353585Z 
2022-12-01T10:48:20.0353834Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0353952Z Ran 1 test in 3.973s
2022-12-01T10:48:20.0353972Z 
2022-12-01T10:48:20.0354065Z OK
2022-12-01T10:48:20.0354084Z 
2022-12-01T10:48:20.0354209Z Generating XML reports...
2022-12-01T10:48:20.0354722Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104103.xml
2022-12-01T10:48:20.0355111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0355289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0355668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0355840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0355881Z 
2022-12-01T10:48:20.0355972Z Running tests...
2022-12-01T10:48:20.0356236Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0356553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0356824Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0357055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51538
2022-12-01T10:48:20.0357277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51539
2022-12-01T10:48:20.0357651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0357828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0358195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0358391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0358761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0358938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0359317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0359575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0359823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0360069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0360454Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0360855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0361087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0361316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0361662Z STAGE:2022-12-01 10:41:13 51538:51538 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0361992Z STAGE:2022-12-01 10:41:13 51539:51539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0362331Z STAGE:2022-12-01 10:41:13 51538:51538 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0362662Z STAGE:2022-12-01 10:41:13 51539:51539 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0362767Z ok (4.177s)
2022-12-01T10:48:20.0362787Z 
2022-12-01T10:48:20.0363034Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0363150Z Ran 1 test in 4.177s
2022-12-01T10:48:20.0363169Z 
2022-12-01T10:48:20.0363265Z OK
2022-12-01T10:48:20.0363284Z 
2022-12-01T10:48:20.0363410Z Generating XML reports...
2022-12-01T10:48:20.0363862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104109.xml
2022-12-01T10:48:20.0364284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0364469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0364856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0365050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0365070Z 
2022-12-01T10:48:20.0365161Z Running tests...
2022-12-01T10:48:20.0365429Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0365747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0366011Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0366232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51829
2022-12-01T10:48:20.0366456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51830
2022-12-01T10:48:20.0366831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0367008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0367369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0367563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0367930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0368107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0368484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0368674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0368979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0369222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0369631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0370009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0370240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0370469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0370804Z STAGE:2022-12-01 10:41:19 51830:51830 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0371129Z STAGE:2022-12-01 10:41:19 51829:51829 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0371471Z STAGE:2022-12-01 10:41:19 51829:51829 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0371802Z STAGE:2022-12-01 10:41:19 51830:51830 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0371908Z ok (4.046s)
2022-12-01T10:48:20.0371927Z 
2022-12-01T10:48:20.0372193Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0372290Z Ran 1 test in 4.047s
2022-12-01T10:48:20.0372309Z 
2022-12-01T10:48:20.0372403Z OK
2022-12-01T10:48:20.0372422Z 
2022-12-01T10:48:20.0372548Z Generating XML reports...
2022-12-01T10:48:20.0373000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104116.xml
2022-12-01T10:48:20.0373370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0373549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0373974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0374177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0374196Z 
2022-12-01T10:48:20.0374287Z Running tests...
2022-12-01T10:48:20.0374559Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0374875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0375160Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0375381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52093
2022-12-01T10:48:20.0375599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52094
2022-12-01T10:48:20.0375976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0376158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0376519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0376714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0377087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0377264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0377638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0377829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0378076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0378432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0378842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0379219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0379451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0379681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0379922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0380165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0380567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0380962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0381204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:20.0381444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:20.0381811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0382200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0382436Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:48:20.0382541Z ok (22.766s)
2022-12-01T10:48:20.0382565Z 
2022-12-01T10:48:20.0382838Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0383043Z Ran 1 test in 22.766s
2022-12-01T10:48:20.0383064Z 
2022-12-01T10:48:20.0383164Z OK
2022-12-01T10:48:20.0383183Z 
2022-12-01T10:48:20.0383313Z Generating XML reports...
2022-12-01T10:48:20.0383770Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104122.xml
2022-12-01T10:48:20.0384123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0384305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0384687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0384880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0384900Z 
2022-12-01T10:48:20.0385011Z Running tests...
2022-12-01T10:48:20.0385282Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0385603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0385905Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0386127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52662
2022-12-01T10:48:20.0386329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52663
2022-12-01T10:48:20.0386702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0386878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0387256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0387498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0387873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0388051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0388430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0388603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0388851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0389096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0389723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0390137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0390377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0390620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0390848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0391083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0391458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0391854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0392100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:20.0392417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:20.0392821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0393215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.0393451Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:48:20.0393682Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:48:20.0393788Z ok (22.374s)
2022-12-01T10:48:20.0393809Z 
2022-12-01T10:48:20.0394056Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0394171Z Ran 1 test in 22.374s
2022-12-01T10:48:20.0394190Z 
2022-12-01T10:48:20.0394288Z OK
2022-12-01T10:48:20.0394308Z 
2022-12-01T10:48:20.0394436Z Generating XML reports...
2022-12-01T10:48:20.0394895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104147.xml
2022-12-01T10:48:20.0395269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0395451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0395835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0396029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0396049Z 
2022-12-01T10:48:20.0396142Z Running tests...
2022-12-01T10:48:20.0396406Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0396722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0397004Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0397300Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53106
2022-12-01T10:48:20.0397520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53107
2022-12-01T10:48:20.0397897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0398076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0398434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0398629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0398998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0399174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0399564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0399757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0400004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0400250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0400654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0401034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0401266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0401497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0401664Z skip: Skipped due to small world size. (3.744s)
2022-12-01T10:48:20.0401726Z 
2022-12-01T10:48:20.0402002Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0402115Z Ran 1 test in 3.744s
2022-12-01T10:48:20.0402135Z 
2022-12-01T10:48:20.0402244Z OK (skipped=1)
2022-12-01T10:48:20.0402263Z 
2022-12-01T10:48:20.0402389Z Generating XML reports...
2022-12-01T10:48:20.0402820Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104212.xml
2022-12-01T10:48:20.0403193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0403375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0403756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0403952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0403972Z 
2022-12-01T10:48:20.0404085Z Running tests...
2022-12-01T10:48:20.0404353Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0404670Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0404937Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0405138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53322
2022-12-01T10:48:20.0405357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53323
2022-12-01T10:48:20.0405733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0405911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0406293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0406553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0406925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0407099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0407460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0407653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0407899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0408144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0408543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0408949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0409186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0409419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0409650Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-12-01T10:48:20.0409734Z ok (5.959s)
2022-12-01T10:48:20.0409775Z 
2022-12-01T10:48:20.0410025Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0410139Z Ran 1 test in 5.959s
2022-12-01T10:48:20.0410158Z 
2022-12-01T10:48:20.0410252Z OK
2022-12-01T10:48:20.0410271Z 
2022-12-01T10:48:20.0410396Z Generating XML reports...
2022-12-01T10:48:20.0410854Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104217.xml
2022-12-01T10:48:20.0411276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0411463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0411846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0412019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0412039Z 
2022-12-01T10:48:20.0412151Z Running tests...
2022-12-01T10:48:20.0412419Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0412735Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0413020Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0413247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53481
2022-12-01T10:48:20.0413473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53482
2022-12-01T10:48:20.0413846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0414002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0414384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0414575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0414945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0415120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0415497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0415744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0415990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0416240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0416624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0417023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0417254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0417482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0417723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0417971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0418370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0418593Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-12-01T10:48:20.0418771Z No ranks successfully processed in monitoredBarrier.
2022-12-01T10:48:20.0418979Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-12-01T10:48:20.0419378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0419483Z ok (3.892s)
2022-12-01T10:48:20.0419502Z 
2022-12-01T10:48:20.0419770Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0419882Z Ran 1 test in 3.893s
2022-12-01T10:48:20.0419905Z 
2022-12-01T10:48:20.0420000Z OK
2022-12-01T10:48:20.0420019Z 
2022-12-01T10:48:20.0420186Z Generating XML reports...
2022-12-01T10:48:20.0420653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104225.xml
2022-12-01T10:48:20.0421008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0421186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0421566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0421759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0421779Z 
2022-12-01T10:48:20.0421889Z Running tests...
2022-12-01T10:48:20.0422151Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0422465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0422753Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0422976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53706
2022-12-01T10:48:20.0423175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53707
2022-12-01T10:48:20.0423548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0423723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0424103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0424294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0424656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0424885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0425262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0425431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0425680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0425925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0426325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0426720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0426952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0427189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0427491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0427732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0428111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0428507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0428740Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T10:48:20.0428844Z ok (4.187s)
2022-12-01T10:48:20.0428865Z 
2022-12-01T10:48:20.0429132Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0429253Z Ran 1 test in 4.187s
2022-12-01T10:48:20.0429273Z 
2022-12-01T10:48:20.0429694Z OK
2022-12-01T10:48:20.0429791Z 
2022-12-01T10:48:20.0429947Z Generating XML reports...
2022-12-01T10:48:20.0430418Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104232.xml
2022-12-01T10:48:20.0430771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0430952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0431336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0431529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0431549Z 
2022-12-01T10:48:20.0431661Z Running tests...
2022-12-01T10:48:20.0431931Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0432254Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0432538Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0432739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53929
2022-12-01T10:48:20.0432962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53930
2022-12-01T10:48:20.0433334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0433668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0434059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0434254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0434624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0434880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0435264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0435437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0435684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0435929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0436330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0436727Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0436958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0437197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0437359Z skip: Skipped due to small world size. (3.861s)
2022-12-01T10:48:20.0437380Z 
2022-12-01T10:48:20.0437649Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0437757Z Ran 1 test in 3.861s
2022-12-01T10:48:20.0437783Z 
2022-12-01T10:48:20.0437997Z OK (skipped=1)
2022-12-01T10:48:20.0438028Z 
2022-12-01T10:48:20.0438223Z Generating XML reports...
2022-12-01T10:48:20.0438692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104238.xml
2022-12-01T10:48:20.0439063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0439240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0439681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0439884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0439904Z 
2022-12-01T10:48:20.0439995Z Running tests...
2022-12-01T10:48:20.0440263Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0440579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0440983Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:48:20.0441004Z 
2022-12-01T10:48:20.0441266Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0441381Z Ran 1 test in 0.003s
2022-12-01T10:48:20.0441401Z 
2022-12-01T10:48:20.0441512Z OK (skipped=1)
2022-12-01T10:48:20.0441531Z 
2022-12-01T10:48:20.0441658Z Generating XML reports...
2022-12-01T10:48:20.0442114Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104244.xml
2022-12-01T10:48:20.0442469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0442647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0443028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0443221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0443241Z 
2022-12-01T10:48:20.0443351Z Running tests...
2022-12-01T10:48:20.0443614Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0443930Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0444337Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:48:20.0444409Z 
2022-12-01T10:48:20.0444686Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0444782Z Ran 1 test in 0.003s
2022-12-01T10:48:20.0444802Z 
2022-12-01T10:48:20.0444920Z OK (skipped=1)
2022-12-01T10:48:20.0444940Z 
2022-12-01T10:48:20.0445069Z Generating XML reports...
2022-12-01T10:48:20.0445523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104246.xml
2022-12-01T10:48:20.0445896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0446074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0446457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0446650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0446673Z 
2022-12-01T10:48:20.0446787Z Running tests...
2022-12-01T10:48:20.0447034Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0447349Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0447751Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T10:48:20.0447771Z 
2022-12-01T10:48:20.0448034Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0448150Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0448170Z 
2022-12-01T10:48:20.0448279Z OK (skipped=1)
2022-12-01T10:48:20.0448298Z 
2022-12-01T10:48:20.0448422Z Generating XML reports...
2022-12-01T10:48:20.0448866Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104248.xml
2022-12-01T10:48:20.0449265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0449452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0449833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0450026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0450046Z 
2022-12-01T10:48:20.0450157Z Running tests...
2022-12-01T10:48:20.0450421Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0450736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0451129Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-12-01T10:48:20.0451149Z 
2022-12-01T10:48:20.0451410Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0451508Z Ran 1 test in 0.003s
2022-12-01T10:48:20.0451548Z 
2022-12-01T10:48:20.0451642Z OK (skipped=1)
2022-12-01T10:48:20.0451661Z 
2022-12-01T10:48:20.0451786Z Generating XML reports...
2022-12-01T10:48:20.0452231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104250.xml
2022-12-01T10:48:20.0452604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0452780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0453160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0453354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0453373Z 
2022-12-01T10:48:20.0453485Z Running tests...
2022-12-01T10:48:20.0453731Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0454109Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0454411Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-12-01T10:48:20.0454430Z 
2022-12-01T10:48:20.0454695Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0454809Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0454829Z 
2022-12-01T10:48:20.0454938Z OK (skipped=1)
2022-12-01T10:48:20.0454957Z 
2022-12-01T10:48:20.0455080Z Generating XML reports...
2022-12-01T10:48:20.0455531Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104252.xml
2022-12-01T10:48:20.0455908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0456068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0456460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0456653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0456672Z 
2022-12-01T10:48:20.0456782Z Running tests...
2022-12-01T10:48:20.0457048Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0457361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0457612Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:48:20.0457632Z 
2022-12-01T10:48:20.0457895Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0457988Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0458029Z 
2022-12-01T10:48:20.0458118Z OK (skipped=1)
2022-12-01T10:48:20.0458137Z 
2022-12-01T10:48:20.0458268Z Generating XML reports...
2022-12-01T10:48:20.0458762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104254.xml
2022-12-01T10:48:20.0459148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0459328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0459711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0459904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0459923Z 
2022-12-01T10:48:20.0460033Z Running tests...
2022-12-01T10:48:20.0460282Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0460596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0460866Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:48:20.0460892Z 
2022-12-01T10:48:20.0461155Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0461269Z Ran 1 test in 0.003s
2022-12-01T10:48:20.0461288Z 
2022-12-01T10:48:20.0461399Z OK (skipped=1)
2022-12-01T10:48:20.0461417Z 
2022-12-01T10:48:20.0461543Z Generating XML reports...
2022-12-01T10:48:20.0461992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104256.xml
2022-12-01T10:48:20.0462365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0462523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0462908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0463100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0463165Z 
2022-12-01T10:48:20.0463283Z Running tests...
2022-12-01T10:48:20.0463550Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0463864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0464174Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:48:20.0464194Z 
2022-12-01T10:48:20.0464453Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0464564Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0464584Z 
2022-12-01T10:48:20.0464673Z OK (skipped=1)
2022-12-01T10:48:20.0464692Z 
2022-12-01T10:48:20.0464816Z Generating XML reports...
2022-12-01T10:48:20.0465265Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104258.xml
2022-12-01T10:48:20.0465648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0465826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0466207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0466401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0466421Z 
2022-12-01T10:48:20.0466531Z Running tests...
2022-12-01T10:48:20.0466774Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0467090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0467396Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0467617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54597
2022-12-01T10:48:20.0467887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54598
2022-12-01T10:48:20.0468273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0468451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0468833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0469027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0469600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0469788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0470176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0470374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0470629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0470878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0471285Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0471684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0471916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0472130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0472235Z ok (3.868s)
2022-12-01T10:48:20.0472255Z 
2022-12-01T10:48:20.0472522Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0472733Z Ran 1 test in 3.868s
2022-12-01T10:48:20.0472755Z 
2022-12-01T10:48:20.0472852Z OK
2022-12-01T10:48:20.0472871Z 
2022-12-01T10:48:20.0473001Z Generating XML reports...
2022-12-01T10:48:20.0473455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104300.xml
2022-12-01T10:48:20.0473829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0473988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0474369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0474561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0474580Z 
2022-12-01T10:48:20.0474690Z Running tests...
2022-12-01T10:48:20.0474956Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0475278Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0475574Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0475796Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54796
2022-12-01T10:48:20.0476014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54797
2022-12-01T10:48:20.0476371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0476548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0476931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0477124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0477550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0477736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0478116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0478354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0478585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0478833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0479239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0479641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0479882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0480112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0480216Z ok (3.833s)
2022-12-01T10:48:20.0480235Z 
2022-12-01T10:48:20.0480508Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0480621Z Ran 1 test in 3.834s
2022-12-01T10:48:20.0480641Z 
2022-12-01T10:48:20.0480716Z OK
2022-12-01T10:48:20.0480735Z 
2022-12-01T10:48:20.0480860Z Generating XML reports...
2022-12-01T10:48:20.0481313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104306.xml
2022-12-01T10:48:20.0481686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0481865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0482313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0482508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0482527Z 
2022-12-01T10:48:20.0482637Z Running tests...
2022-12-01T10:48:20.0482906Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0483202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0483484Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:48:20.0483503Z 
2022-12-01T10:48:20.0483769Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0483883Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0483902Z 
2022-12-01T10:48:20.0484013Z OK (skipped=1)
2022-12-01T10:48:20.0484032Z 
2022-12-01T10:48:20.0484163Z Generating XML reports...
2022-12-01T10:48:20.0484615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104312.xml
2022-12-01T10:48:20.0484990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0485148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0485532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0485726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0485746Z 
2022-12-01T10:48:20.0485856Z Running tests...
2022-12-01T10:48:20.0486123Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0486438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0486737Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T10:48:20.0486804Z 
2022-12-01T10:48:20.0487079Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0487193Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0487213Z 
2022-12-01T10:48:20.0487303Z OK (skipped=1)
2022-12-01T10:48:20.0487342Z 
2022-12-01T10:48:20.0487448Z Generating XML reports...
2022-12-01T10:48:20.0487898Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104314.xml
2022-12-01T10:48:20.0488269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0488448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0488827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0489029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0489049Z 
2022-12-01T10:48:20.0489162Z Running tests...
2022-12-01T10:48:20.0489429Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0489726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0490006Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0490761Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2022-12-01T10:48:20.0490782Z 
2022-12-01T10:48:20.0491048Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0491162Z Ran 1 test in 1.665s
2022-12-01T10:48:20.0491237Z 
2022-12-01T10:48:20.0491353Z OK (skipped=1)
2022-12-01T10:48:20.0491372Z 
2022-12-01T10:48:20.0491504Z Generating XML reports...
2022-12-01T10:48:20.0491958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104316.xml
2022-12-01T10:48:20.0492332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0492512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0492878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0493074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0493093Z 
2022-12-01T10:48:20.0493204Z Running tests...
2022-12-01T10:48:20.0493471Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0493787Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0494074Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0494298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55195
2022-12-01T10:48:20.0494518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55196
2022-12-01T10:48:20.0494869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0495051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0495432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0495626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0495991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0496217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0496607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0496801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0497050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0497280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0497681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0498080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0498311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0498548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0498812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjmg0aiop
2022-12-01T10:48:20.0499086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjmg0aiop/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0499343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6qz3sksd
2022-12-01T10:48:20.0499611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6qz3sksd/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0499697Z ok (5.976s)
2022-12-01T10:48:20.0499716Z 
2022-12-01T10:48:20.0499987Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0500100Z Ran 1 test in 5.976s
2022-12-01T10:48:20.0500120Z 
2022-12-01T10:48:20.0500214Z OK
2022-12-01T10:48:20.0500233Z 
2022-12-01T10:48:20.0500412Z Generating XML reports...
2022-12-01T10:48:20.0500870Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104320.xml
2022-12-01T10:48:20.0501243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0501423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0501786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0501979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0501999Z 
2022-12-01T10:48:20.0502110Z Running tests...
2022-12-01T10:48:20.0502374Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0502686Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0502959Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0503189Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55442
2022-12-01T10:48:20.0503407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55443
2022-12-01T10:48:20.0503761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0503943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0504321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0504514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0504876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0505052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0505480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0505678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0505927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0506152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0506561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0506959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0507192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0507423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0507532Z ok (6.544s)
2022-12-01T10:48:20.0507551Z 
2022-12-01T10:48:20.0507824Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0507941Z Ran 1 test in 6.544s
2022-12-01T10:48:20.0507960Z 
2022-12-01T10:48:20.0508055Z OK
2022-12-01T10:48:20.0508074Z 
2022-12-01T10:48:20.0508180Z Generating XML reports...
2022-12-01T10:48:20.0508637Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104328.xml
2022-12-01T10:48:20.0509009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0509187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0509799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0509996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0510094Z 
2022-12-01T10:48:20.0510214Z Running tests...
2022-12-01T10:48:20.0510490Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0510788Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0511077Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0511299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55719
2022-12-01T10:48:20.0511517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55720
2022-12-01T10:48:20.0511885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0512061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0512442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0512642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0513009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0513167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0513543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0513731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0513979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0514225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0514624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0515073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0515320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0515549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0515633Z ok (6.612s)
2022-12-01T10:48:20.0515653Z 
2022-12-01T10:48:20.0515925Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0516038Z Ran 1 test in 6.612s
2022-12-01T10:48:20.0516058Z 
2022-12-01T10:48:20.0516153Z OK
2022-12-01T10:48:20.0516172Z 
2022-12-01T10:48:20.0516299Z Generating XML reports...
2022-12-01T10:48:20.0516751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104337.xml
2022-12-01T10:48:20.0517121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0517306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0517671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0517865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0517885Z 
2022-12-01T10:48:20.0517997Z Running tests...
2022-12-01T10:48:20.0518258Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0518572Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0518851Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0519606Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.678s)
2022-12-01T10:48:20.0519675Z 
2022-12-01T10:48:20.0519949Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0520063Z Ran 1 test in 1.678s
2022-12-01T10:48:20.0520082Z 
2022-12-01T10:48:20.0520192Z OK (skipped=1)
2022-12-01T10:48:20.0520211Z 
2022-12-01T10:48:20.0520317Z Generating XML reports...
2022-12-01T10:48:20.0520766Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104345.xml
2022-12-01T10:48:20.0521138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0521317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0521698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0521901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0521920Z 
2022-12-01T10:48:20.0522032Z Running tests...
2022-12-01T10:48:20.0522302Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0522597Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0522894Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0523640Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.742s)
2022-12-01T10:48:20.0523660Z 
2022-12-01T10:48:20.0523924Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0524042Z Ran 1 test in 1.742s
2022-12-01T10:48:20.0524061Z 
2022-12-01T10:48:20.0524213Z OK (skipped=1)
2022-12-01T10:48:20.0524234Z 
2022-12-01T10:48:20.0524365Z Generating XML reports...
2022-12-01T10:48:20.0524823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104349.xml
2022-12-01T10:48:20.0525194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0525373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0525739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0525932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0525952Z 
2022-12-01T10:48:20.0526061Z Running tests...
2022-12-01T10:48:20.0526328Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0526652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0526965Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0527186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56139
2022-12-01T10:48:20.0527405Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56140
2022-12-01T10:48:20.0527778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0527935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0528317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0528510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0528972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0529152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0529531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0529722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0529969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0530199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0530602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0531004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0531243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0531474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0531629Z skip: Need at least 4 CUDA devices (3.717s)
2022-12-01T10:48:20.0531649Z 
2022-12-01T10:48:20.0531915Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0532029Z Ran 1 test in 3.717s
2022-12-01T10:48:20.0532048Z 
2022-12-01T10:48:20.0532157Z OK (skipped=1)
2022-12-01T10:48:20.0532176Z 
2022-12-01T10:48:20.0532282Z Generating XML reports...
2022-12-01T10:48:20.0532738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104353.xml
2022-12-01T10:48:20.0533109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0533285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0533729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0533934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0533953Z 
2022-12-01T10:48:20.0534062Z Running tests...
2022-12-01T10:48:20.0534326Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0534622Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0534953Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0535176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56305
2022-12-01T10:48:20.0535394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56306
2022-12-01T10:48:20.0535766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0535951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0536340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0536534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0536899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0537054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0537433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0537623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0537872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0538189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0538595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0538999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0539231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0539462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0539594Z skip: Need at least 4 CUDA devices (3.688s)
2022-12-01T10:48:20.0539613Z 
2022-12-01T10:48:20.0539885Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0540001Z Ran 1 test in 3.688s
2022-12-01T10:48:20.0540020Z 
2022-12-01T10:48:20.0540134Z OK (skipped=1)
2022-12-01T10:48:20.0540154Z 
2022-12-01T10:48:20.0540279Z Generating XML reports...
2022-12-01T10:48:20.0540738Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104359.xml
2022-12-01T10:48:20.0541118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0541294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0541676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0541849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0541869Z 
2022-12-01T10:48:20.0541982Z Running tests...
2022-12-01T10:48:20.0542247Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0542562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0542900Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0543660Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.565s)
2022-12-01T10:48:20.0543680Z 
2022-12-01T10:48:20.0543944Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0544056Z Ran 1 test in 1.565s
2022-12-01T10:48:20.0544075Z 
2022-12-01T10:48:20.0544184Z OK (skipped=1)
2022-12-01T10:48:20.0544203Z 
2022-12-01T10:48:20.0544310Z Generating XML reports...
2022-12-01T10:48:20.0544760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104405.xml
2022-12-01T10:48:20.0545147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0545326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0545707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0545899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0545918Z 
2022-12-01T10:48:20.0546031Z Running tests...
2022-12-01T10:48:20.0546296Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0546611Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0546860Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0547082Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56612
2022-12-01T10:48:20.0547358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56613
2022-12-01T10:48:20.0547737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0547917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0548301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0548497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0548866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0549021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0549623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0549828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0550089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0550339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0550756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0551158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0551393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0551617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0551839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0552071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0552527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0552926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0553252Z STAGE:2022-12-01 10:44:12 56613:56613 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0553575Z STAGE:2022-12-01 10:44:12 56612:56612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0553900Z STAGE:2022-12-01 10:44:12 56613:56613 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0554217Z STAGE:2022-12-01 10:44:12 56612:56612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0554541Z STAGE:2022-12-01 10:44:12 56613:56613 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0554853Z STAGE:2022-12-01 10:44:12 56612:56612 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0555195Z STAGE:2022-12-01 10:44:12 56612:56612 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0555525Z STAGE:2022-12-01 10:44:12 56613:56613 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0555632Z ok (3.963s)
2022-12-01T10:48:20.0555652Z 
2022-12-01T10:48:20.0555920Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0556039Z Ran 1 test in 3.963s
2022-12-01T10:48:20.0556059Z 
2022-12-01T10:48:20.0556154Z OK
2022-12-01T10:48:20.0556173Z 
2022-12-01T10:48:20.0556301Z Generating XML reports...
2022-12-01T10:48:20.0556755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104408.xml
2022-12-01T10:48:20.0557114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0557371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0557756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0557953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0557972Z 
2022-12-01T10:48:20.0558084Z Running tests...
2022-12-01T10:48:20.0558351Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0558669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0558936Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0559140Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56838
2022-12-01T10:48:20.0559359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56839
2022-12-01T10:48:20.0559743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0559924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0560306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0560500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0560865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0561043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0561422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0561596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0561848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0562138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0562553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0562951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0563184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0563418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0563662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0563908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0564295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0564692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0565032Z STAGE:2022-12-01 10:44:18 56838:56838 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0565368Z STAGE:2022-12-01 10:44:18 56839:56839 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0565712Z STAGE:2022-12-01 10:44:18 56839:56839 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0566044Z STAGE:2022-12-01 10:44:18 56838:56838 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0566371Z STAGE:2022-12-01 10:44:18 56839:56839 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0566691Z STAGE:2022-12-01 10:44:18 56838:56838 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0567099Z STAGE:2022-12-01 10:44:18 56838:56838 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0567414Z STAGE:2022-12-01 10:44:18 56839:56839 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0567523Z ok (3.897s)
2022-12-01T10:48:20.0567543Z 
2022-12-01T10:48:20.0567810Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0567927Z Ran 1 test in 3.898s
2022-12-01T10:48:20.0567947Z 
2022-12-01T10:48:20.0568043Z OK
2022-12-01T10:48:20.0568062Z 
2022-12-01T10:48:20.0568191Z Generating XML reports...
2022-12-01T10:48:20.0568651Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104414.xml
2022-12-01T10:48:20.0569027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0569190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0569583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0569778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0569797Z 
2022-12-01T10:48:20.0569908Z Running tests...
2022-12-01T10:48:20.0570177Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0570491Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0570768Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0570990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57003
2022-12-01T10:48:20.0571210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57004
2022-12-01T10:48:20.0571566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0571794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0572187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0572381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0572746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0572925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0573301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0573493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0573723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0573978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0574382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0574782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0575018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0575251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0575494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0575739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0576136Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0576575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0576915Z STAGE:2022-12-01 10:44:24 57004:57004 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0577240Z STAGE:2022-12-01 10:44:24 57003:57003 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0577786Z STAGE:2022-12-01 10:44:24 57004:57004 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:44:24 57003:57003 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0577807Z 
2022-12-01T10:48:20.0578136Z STAGE:2022-12-01 10:44:24 57004:57004 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0578511Z STAGE:2022-12-01 10:44:24 57003:57003 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0579067Z STAGE:2022-12-01 10:44:24 57004:57004 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:44:24 57003:57003 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0579088Z 
2022-12-01T10:48:20.0579195Z ok (3.805s)
2022-12-01T10:48:20.0579215Z 
2022-12-01T10:48:20.0579483Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0579601Z Ran 1 test in 3.805s
2022-12-01T10:48:20.0579620Z 
2022-12-01T10:48:20.0579695Z OK
2022-12-01T10:48:20.0579734Z 
2022-12-01T10:48:20.0579844Z Generating XML reports...
2022-12-01T10:48:20.0580295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104421.xml
2022-12-01T10:48:20.0580673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0580853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0581284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0581486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0581507Z 
2022-12-01T10:48:20.0581619Z Running tests...
2022-12-01T10:48:20.0581887Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0582184Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0582451Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0582673Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57228
2022-12-01T10:48:20.0582894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57229
2022-12-01T10:48:20.0583270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0583452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0583833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0584029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0584402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0584564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0584944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0585137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0585384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0585630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0586092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0586493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0586726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0586938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0587183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0587427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0587828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0588229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0588568Z STAGE:2022-12-01 10:44:30 57229:57229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0588898Z STAGE:2022-12-01 10:44:30 57228:57228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0589236Z STAGE:2022-12-01 10:44:30 57228:57228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0589804Z STAGE:2022-12-01 10:44:30 57229:57229 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0590128Z STAGE:2022-12-01 10:44:30 57229:57229 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0590455Z STAGE:2022-12-01 10:44:30 57228:57228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0590791Z STAGE:2022-12-01 10:44:30 57229:57229 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0591193Z STAGE:2022-12-01 10:44:30 57228:57228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0591309Z ok (3.953s)
2022-12-01T10:48:20.0591328Z 
2022-12-01T10:48:20.0591601Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0591718Z Ran 1 test in 3.953s
2022-12-01T10:48:20.0591737Z 
2022-12-01T10:48:20.0591833Z OK
2022-12-01T10:48:20.0591852Z 
2022-12-01T10:48:20.0591980Z Generating XML reports...
2022-12-01T10:48:20.0592420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104427.xml
2022-12-01T10:48:20.0592797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0592979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0593364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0593566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0593585Z 
2022-12-01T10:48:20.0593696Z Running tests...
2022-12-01T10:48:20.0593964Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0594280Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0594523Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0594744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57392
2022-12-01T10:48:20.0594962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57393
2022-12-01T10:48:20.0595338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0595516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0595975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0596172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0596540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0596718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0597074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0597267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0597515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0597762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0598169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0598570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0598803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0599033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0599194Z skip: Skipped due to small world size. (3.903s)
2022-12-01T10:48:20.0599214Z 
2022-12-01T10:48:20.0599464Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0599580Z Ran 1 test in 3.903s
2022-12-01T10:48:20.0599600Z 
2022-12-01T10:48:20.0599710Z OK (skipped=1)
2022-12-01T10:48:20.0599729Z 
2022-12-01T10:48:20.0599855Z Generating XML reports...
2022-12-01T10:48:20.0600309Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104433.xml
2022-12-01T10:48:20.0600731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0600916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0601303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0601478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0601518Z 
2022-12-01T10:48:20.0601610Z Running tests...
2022-12-01T10:48:20.0601879Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0602196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0602457Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0602683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57615
2022-12-01T10:48:20.0602907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57616
2022-12-01T10:48:20.0603280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0603459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0603823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0604017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0604382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0604558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0604942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0605201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0605452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0605701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0606088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0606488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0606723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0606955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0607119Z skip: Skipped due to small world size. (3.946s)
2022-12-01T10:48:20.0607142Z 
2022-12-01T10:48:20.0607416Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0607532Z Ran 1 test in 3.946s
2022-12-01T10:48:20.0607552Z 
2022-12-01T10:48:20.0607663Z OK (skipped=1)
2022-12-01T10:48:20.0607682Z 
2022-12-01T10:48:20.0607808Z Generating XML reports...
2022-12-01T10:48:20.0608244Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104439.xml
2022-12-01T10:48:20.0608620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0608800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0609181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0609376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0609399Z 
2022-12-01T10:48:20.0609513Z Running tests...
2022-12-01T10:48:20.0609822Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0610150Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0610419Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0610622Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57814
2022-12-01T10:48:20.0610842Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57815
2022-12-01T10:48:20.0611215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0611394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0611773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0611970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0612338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0612515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0612870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0613061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0613310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0613556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0613956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0614358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0614651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0614886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0615050Z skip: Skipped due to small world size. (3.935s)
2022-12-01T10:48:20.0615069Z 
2022-12-01T10:48:20.0615319Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0615700Z Ran 1 test in 3.936s
2022-12-01T10:48:20.0615867Z 
2022-12-01T10:48:20.0615983Z OK (skipped=1)
2022-12-01T10:48:20.0616147Z 
2022-12-01T10:48:20.0616279Z Generating XML reports...
2022-12-01T10:48:20.0616873Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104445.xml
2022-12-01T10:48:20.0617603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0618069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0618649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0619112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0619349Z 
2022-12-01T10:48:20.0619465Z Running tests...
2022-12-01T10:48:20.0619877Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0620400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0620916Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0621415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57955
2022-12-01T10:48:20.0621875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57956
2022-12-01T10:48:20.0622529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0622999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0623580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0624036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0624620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0625065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0625636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0626082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0626539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0627084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0627752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0628430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0628948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0629651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0630067Z skip: Skipped due to small world size. (4.614s)
2022-12-01T10:48:20.0630270Z 
2022-12-01T10:48:20.0630530Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0630865Z Ran 1 test in 4.614s
2022-12-01T10:48:20.0631126Z 
2022-12-01T10:48:20.0631239Z OK (skipped=1)
2022-12-01T10:48:20.0631395Z 
2022-12-01T10:48:20.0631535Z Generating XML reports...
2022-12-01T10:48:20.0632153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104451.xml
2022-12-01T10:48:20.0632874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0633320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0633899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0634351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0634583Z 
2022-12-01T10:48:20.0634694Z Running tests...
2022-12-01T10:48:20.0635098Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0635830Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0636338Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0636817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58154
2022-12-01T10:48:20.0637267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58155
2022-12-01T10:48:20.0637865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0638319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0638893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0639343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0639933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0640459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0641054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0641526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0641989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0642502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0643171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0643871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0644418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0644901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0645488Z STAGE:2022-12-01 10:45:01 58154:58154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0646067Z STAGE:2022-12-01 10:45:01 58155:58155 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0646635Z STAGE:2022-12-01 10:45:01 58155:58155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0647219Z STAGE:2022-12-01 10:45:01 58154:58154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0647788Z STAGE:2022-12-01 10:45:01 58155:58155 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0648336Z STAGE:2022-12-01 10:45:01 58154:58154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0648909Z STAGE:2022-12-01 10:45:01 58154:58154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0649555Z STAGE:2022-12-01 10:45:01 58155:58155 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0649901Z ok (3.823s)
2022-12-01T10:48:20.0650033Z 
2022-12-01T10:48:20.0650303Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0650636Z Ran 1 test in 3.823s
2022-12-01T10:48:20.0650803Z 
2022-12-01T10:48:20.0650898Z OK
2022-12-01T10:48:20.0651037Z 
2022-12-01T10:48:20.0651146Z Generating XML reports...
2022-12-01T10:48:20.0651763Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104458.xml
2022-12-01T10:48:20.0652506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0652974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0653581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0654070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0654301Z 
2022-12-01T10:48:20.0654413Z Running tests...
2022-12-01T10:48:20.0654817Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0655334Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0655829Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0656306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58267
2022-12-01T10:48:20.0656740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58268
2022-12-01T10:48:20.0657348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0657796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0658422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0658886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0659478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0659929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0660530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0661002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0661468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0661999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0662667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0663345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0663877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0664359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0664951Z STAGE:2022-12-01 10:45:08 58268:58268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0665511Z STAGE:2022-12-01 10:45:08 58267:58267 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0666308Z STAGE:2022-12-01 10:45:08 58268:58268 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:45:08 58267:58267 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0666758Z 
2022-12-01T10:48:20.0667098Z STAGE:2022-12-01 10:45:08 58268:58268 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0667670Z STAGE:2022-12-01 10:45:08 58267:58267 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0668227Z STAGE:2022-12-01 10:45:08 58267:58267 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0668802Z STAGE:2022-12-01 10:45:08 58268:58268 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0669151Z ok (4.905s)
2022-12-01T10:48:20.0669300Z 
2022-12-01T10:48:20.0669796Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0670131Z Ran 1 test in 4.905s
2022-12-01T10:48:20.0670292Z 
2022-12-01T10:48:20.0670389Z OK
2022-12-01T10:48:20.0670524Z 
2022-12-01T10:48:20.0670649Z Generating XML reports...
2022-12-01T10:48:20.0671246Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104504.xml
2022-12-01T10:48:20.0671975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0672429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0672989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0673463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0673697Z 
2022-12-01T10:48:20.0673806Z Running tests...
2022-12-01T10:48:20.0674215Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0674736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0675260Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.004s)
2022-12-01T10:48:20.0675565Z 
2022-12-01T10:48:20.0675836Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0676237Z Ran 1 test in 0.004s
2022-12-01T10:48:20.0676395Z 
2022-12-01T10:48:20.0676506Z OK (skipped=1)
2022-12-01T10:48:20.0676762Z 
2022-12-01T10:48:20.0676888Z Generating XML reports...
2022-12-01T10:48:20.0677585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104511.xml
2022-12-01T10:48:20.0678314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0678770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0679348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0679820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0680033Z 
2022-12-01T10:48:20.0680144Z Running tests...
2022-12-01T10:48:20.0680553Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0681087Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0681571Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0682055Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58413
2022-12-01T10:48:20.0682506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58414
2022-12-01T10:48:20.0683112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0683544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0684123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0684590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0685245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0685691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0686259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0686722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0687157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0687657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0688311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0689005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0689519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0690111Z STAGE:2022-12-01 10:45:16 58414:58414 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0690587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0691146Z STAGE:2022-12-01 10:45:16 58413:58413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0691735Z STAGE:2022-12-01 10:45:16 58414:58414 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0692319Z STAGE:2022-12-01 10:45:16 58413:58413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0692886Z STAGE:2022-12-01 10:45:16 58414:58414 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0693427Z STAGE:2022-12-01 10:45:16 58413:58413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0694059Z STAGE:2022-12-01 10:45:16 58413:58413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0694647Z STAGE:2022-12-01 10:45:16 58414:58414 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0694990Z ok (3.789s)
2022-12-01T10:48:20.0695121Z 
2022-12-01T10:48:20.0695387Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0695713Z Ran 1 test in 3.789s
2022-12-01T10:48:20.0695876Z 
2022-12-01T10:48:20.0695971Z OK
2022-12-01T10:48:20.0696106Z 
2022-12-01T10:48:20.0696215Z Generating XML reports...
2022-12-01T10:48:20.0696819Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104513.xml
2022-12-01T10:48:20.0697532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0697982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0698549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0699020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0699249Z 
2022-12-01T10:48:20.0699359Z Running tests...
2022-12-01T10:48:20.0699745Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0700275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0700813Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-12-01T10:48:20.0701134Z 
2022-12-01T10:48:20.0701400Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0701709Z Ran 1 test in 0.003s
2022-12-01T10:48:20.0701871Z 
2022-12-01T10:48:20.0701980Z OK (skipped=1)
2022-12-01T10:48:20.0702202Z 
2022-12-01T10:48:20.0702329Z Generating XML reports...
2022-12-01T10:48:20.0702930Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104518.xml
2022-12-01T10:48:20.0703640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0704086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0704657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0705111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0705341Z 
2022-12-01T10:48:20.0705451Z Running tests...
2022-12-01T10:48:20.0705857Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0706368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0706899Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-12-01T10:48:20.0707202Z 
2022-12-01T10:48:20.0707470Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0707798Z Ran 1 test in 0.003s
2022-12-01T10:48:20.0707958Z 
2022-12-01T10:48:20.0708049Z OK (skipped=1)
2022-12-01T10:48:20.0708202Z 
2022-12-01T10:48:20.0708326Z Generating XML reports...
2022-12-01T10:48:20.0708929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104520.xml
2022-12-01T10:48:20.0709891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0710330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0710907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0711380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0711610Z 
2022-12-01T10:48:20.0711780Z Running tests...
2022-12-01T10:48:20.0712202Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0712733Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0713226Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0713689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58592
2022-12-01T10:48:20.0714139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58593
2022-12-01T10:48:20.0714748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0715178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0715753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0716232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0716814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0717241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0717814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0718280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0718716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0719215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0719872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0720658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0721162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0721744Z STAGE:2022-12-01 10:45:26 58593:58593 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0722221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0722801Z STAGE:2022-12-01 10:45:26 58592:58592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0723578Z STAGE:2022-12-01 10:45:26 58592:58592 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:45:26 58593:58593 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0723966Z 
2022-12-01T10:48:20.0724296Z STAGE:2022-12-01 10:45:26 58593:58593 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0724871Z STAGE:2022-12-01 10:45:26 58592:58592 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0725658Z STAGE:2022-12-01 10:45:26 58593:58593 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:45:26 58592:58592 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0726044Z 
2022-12-01T10:48:20.0726132Z ok (3.788s)
2022-12-01T10:48:20.0726279Z 
2022-12-01T10:48:20.0726547Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0726876Z Ran 1 test in 3.788s
2022-12-01T10:48:20.0727041Z 
2022-12-01T10:48:20.0727133Z OK
2022-12-01T10:48:20.0727249Z 
2022-12-01T10:48:20.0727428Z Generating XML reports...
2022-12-01T10:48:20.0728045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104522.xml
2022-12-01T10:48:20.0728756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0729283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0729872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0730345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0730572Z 
2022-12-01T10:48:20.0730681Z Running tests...
2022-12-01T10:48:20.0731069Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0731598Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0732099Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:48:20.0732387Z 
2022-12-01T10:48:20.0732652Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0732967Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0733129Z 
2022-12-01T10:48:20.0733243Z OK (skipped=1)
2022-12-01T10:48:20.0733397Z 
2022-12-01T10:48:20.0733523Z Generating XML reports...
2022-12-01T10:48:20.0734115Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104528.xml
2022-12-01T10:48:20.0734827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0735272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0735847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0736303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0736531Z 
2022-12-01T10:48:20.0736641Z Running tests...
2022-12-01T10:48:20.0737044Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0737627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0738141Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T10:48:20.0738436Z 
2022-12-01T10:48:20.0738700Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0739021Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0739166Z 
2022-12-01T10:48:20.0739276Z OK (skipped=1)
2022-12-01T10:48:20.0739431Z 
2022-12-01T10:48:20.0739556Z Generating XML reports...
2022-12-01T10:48:20.0740161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104530.xml
2022-12-01T10:48:20.0740852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0741302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0741882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0742352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0742563Z 
2022-12-01T10:48:20.0742674Z Running tests...
2022-12-01T10:48:20.0743073Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0743599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0744087Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0744579Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58771
2022-12-01T10:48:20.0745033Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58772
2022-12-01T10:48:20.0745640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0746077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0746703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0747179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0747746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0748194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0748782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0749244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0749923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0750423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0751101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0751803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0752308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0752893Z STAGE:2022-12-01 10:45:35 58772:58772 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0753371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0753951Z STAGE:2022-12-01 10:45:35 58771:58771 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0754519Z STAGE:2022-12-01 10:45:35 58772:58772 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0755096Z STAGE:2022-12-01 10:45:35 58771:58771 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0755770Z STAGE:2022-12-01 10:45:35 58772:58772 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0756319Z STAGE:2022-12-01 10:45:35 58771:58771 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0756891Z STAGE:2022-12-01 10:45:35 58771:58771 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0757471Z STAGE:2022-12-01 10:45:35 58772:58772 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0757815Z ok (3.881s)
2022-12-01T10:48:20.0757946Z 
2022-12-01T10:48:20.0758215Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0758545Z Ran 1 test in 3.882s
2022-12-01T10:48:20.0758707Z 
2022-12-01T10:48:20.0758802Z OK
2022-12-01T10:48:20.0758942Z 
2022-12-01T10:48:20.0759050Z Generating XML reports...
2022-12-01T10:48:20.0759657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104532.xml
2022-12-01T10:48:20.0760376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0760826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0761381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0761849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0762079Z 
2022-12-01T10:48:20.0762188Z Running tests...
2022-12-01T10:48:20.0762574Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0763102Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0763591Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0764067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58884
2022-12-01T10:48:20.0764566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58885
2022-12-01T10:48:20.0765189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0765638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0766191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0766663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0767240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0767685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0768237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0768713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0769161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0769658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0770300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0770988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0771508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0771965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0772548Z STAGE:2022-12-01 10:45:41 58885:58885 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0773186Z STAGE:2022-12-01 10:45:41 58884:58884 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0773770Z STAGE:2022-12-01 10:45:41 58884:58884 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0774337Z STAGE:2022-12-01 10:45:41 58885:58885 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0774913Z STAGE:2022-12-01 10:45:41 58884:58884 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0775476Z STAGE:2022-12-01 10:45:41 58885:58885 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0776049Z STAGE:2022-12-01 10:45:41 58885:58885 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0776609Z STAGE:2022-12-01 10:45:41 58884:58884 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0776958Z ok (3.654s)
2022-12-01T10:48:20.0777113Z 
2022-12-01T10:48:20.0777384Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0777697Z Ran 1 test in 3.654s
2022-12-01T10:48:20.0777859Z 
2022-12-01T10:48:20.0777951Z OK
2022-12-01T10:48:20.0778085Z 
2022-12-01T10:48:20.0778210Z Generating XML reports...
2022-12-01T10:48:20.0778878Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104538.xml
2022-12-01T10:48:20.0779582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0780032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0780606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0781058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0781289Z 
2022-12-01T10:48:20.0781403Z Running tests...
2022-12-01T10:48:20.0781859Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0782404Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0782885Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0783365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58997
2022-12-01T10:48:20.0783820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58998
2022-12-01T10:48:20.0784412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0784858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0785429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0785892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0786459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0786907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0787481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0787945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0788378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0788876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0789745Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0790431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0791042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0791517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0791864Z ok (3.568s)
2022-12-01T10:48:20.0791994Z 
2022-12-01T10:48:20.0792271Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0792597Z Ran 1 test in 3.568s
2022-12-01T10:48:20.0792759Z 
2022-12-01T10:48:20.0792853Z OK
2022-12-01T10:48:20.0792988Z 
2022-12-01T10:48:20.0793095Z Generating XML reports...
2022-12-01T10:48:20.0793703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104543.xml
2022-12-01T10:48:20.0794412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0794864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0795430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0795898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0796126Z 
2022-12-01T10:48:20.0796236Z Running tests...
2022-12-01T10:48:20.0796618Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0797143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0797646Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0798132Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59106
2022-12-01T10:48:20.0798565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59107
2022-12-01T10:48:20.0799169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0799686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0800268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0800741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0801315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0801755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0802305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0802766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0803214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0803718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0804358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0805043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0805563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0806127Z STAGE:2022-12-01 10:45:52 59107:59107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0806607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0807185Z STAGE:2022-12-01 10:45:53 59106:59106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0807768Z STAGE:2022-12-01 10:45:53 59106:59106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0808418Z STAGE:2022-12-01 10:45:53 59107:59107 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0808991Z STAGE:2022-12-01 10:45:53 59107:59107 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0809556Z STAGE:2022-12-01 10:45:53 59106:59106 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0810129Z STAGE:2022-12-01 10:45:53 59106:59106 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0810692Z STAGE:2022-12-01 10:45:53 59107:59107 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0811044Z ok (3.811s)
2022-12-01T10:48:20.0811192Z 
2022-12-01T10:48:20.0811464Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0811774Z Ran 1 test in 3.811s
2022-12-01T10:48:20.0811937Z 
2022-12-01T10:48:20.0812030Z OK
2022-12-01T10:48:20.0812165Z 
2022-12-01T10:48:20.0812294Z Generating XML reports...
2022-12-01T10:48:20.0812903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104549.xml
2022-12-01T10:48:20.0813606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0814053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0814632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0815085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0815315Z 
2022-12-01T10:48:20.0815425Z Running tests...
2022-12-01T10:48:20.0815830Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0816362Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0816845Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:48:20.0817183Z 
2022-12-01T10:48:20.0817459Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0817789Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0817949Z 
2022-12-01T10:48:20.0818040Z OK (skipped=1)
2022-12-01T10:48:20.0818197Z 
2022-12-01T10:48:20.0818322Z Generating XML reports...
2022-12-01T10:48:20.0818929Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104555.xml
2022-12-01T10:48:20.0819637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0820069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0820645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0821117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0821352Z 
2022-12-01T10:48:20.0821465Z Running tests...
2022-12-01T10:48:20.0821854Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0822387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0822903Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T10:48:20.0823197Z 
2022-12-01T10:48:20.0823443Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0823770Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0823931Z 
2022-12-01T10:48:20.0824037Z OK (skipped=1)
2022-12-01T10:48:20.0824188Z 
2022-12-01T10:48:20.0824311Z Generating XML reports...
2022-12-01T10:48:20.0824902Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104557.xml
2022-12-01T10:48:20.0825615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0826124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0826683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0827151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0827379Z 
2022-12-01T10:48:20.0827487Z Running tests...
2022-12-01T10:48:20.0827893Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0828406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0828910Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0829641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59285
2022-12-01T10:48:20.0830086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59286
2022-12-01T10:48:20.0830707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0831153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0831724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0832179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0832758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0833198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0833766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0834211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0834741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0835255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0835895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0836587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0837105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0837576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0838041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.0838530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.0839191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0839871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.0840479Z STAGE:2022-12-01 10:46:02 59285:59285 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0841051Z STAGE:2022-12-01 10:46:02 59286:59286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0841631Z STAGE:2022-12-01 10:46:02 59286:59286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0842186Z STAGE:2022-12-01 10:46:02 59285:59285 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0842756Z STAGE:2022-12-01 10:46:02 59285:59285 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0843426Z STAGE:2022-12-01 10:46:02 59286:59286 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0844000Z STAGE:2022-12-01 10:46:02 59285:59285 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0844553Z STAGE:2022-12-01 10:46:02 59286:59286 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0844901Z ok (3.913s)
2022-12-01T10:48:20.0845048Z 
2022-12-01T10:48:20.0845316Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0845640Z Ran 1 test in 3.914s
2022-12-01T10:48:20.0845784Z 
2022-12-01T10:48:20.0845875Z OK
2022-12-01T10:48:20.0846007Z 
2022-12-01T10:48:20.0846130Z Generating XML reports...
2022-12-01T10:48:20.0846736Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104559.xml
2022-12-01T10:48:20.0847433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0847889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0848462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0848928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0849139Z 
2022-12-01T10:48:20.0849247Z Running tests...
2022-12-01T10:48:20.0849645Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0850177Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0850659Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0851142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59404
2022-12-01T10:48:20.0851593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59405
2022-12-01T10:48:20.0852251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0852694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0853274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0853742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0854296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0854739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0855307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0855768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0856203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0856706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0857362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0858049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0858554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0859023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0859426Z skip: Skipped due to small world size. (3.697s)
2022-12-01T10:48:20.0859626Z 
2022-12-01T10:48:20.0859875Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0860205Z Ran 1 test in 3.697s
2022-12-01T10:48:20.0860428Z 
2022-12-01T10:48:20.0860535Z OK (skipped=1)
2022-12-01T10:48:20.0860687Z 
2022-12-01T10:48:20.0860813Z Generating XML reports...
2022-12-01T10:48:20.0861405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104605.xml
2022-12-01T10:48:20.0862122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0862571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0863130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0863605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0863833Z 
2022-12-01T10:48:20.0863941Z Running tests...
2022-12-01T10:48:20.0864350Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0864865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0865388Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0865888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59513
2022-12-01T10:48:20.0866321Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59514
2022-12-01T10:48:20.0866938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0867114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0867500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0867692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0868059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0868238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0868669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0868850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0869103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0869572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0870015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0870418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0870651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0870890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0870992Z ok (3.823s)
2022-12-01T10:48:20.0871012Z 
2022-12-01T10:48:20.0871279Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0871376Z Ran 1 test in 3.824s
2022-12-01T10:48:20.0871395Z 
2022-12-01T10:48:20.0871488Z OK
2022-12-01T10:48:20.0871508Z 
2022-12-01T10:48:20.0871637Z Generating XML reports...
2022-12-01T10:48:20.0872090Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104610.xml
2022-12-01T10:48:20.0872468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0872644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0873028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0873309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0873329Z 
2022-12-01T10:48:20.0873420Z Running tests...
2022-12-01T10:48:20.0873691Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0874004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0874252Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0874472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59622
2022-12-01T10:48:20.0874689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59623
2022-12-01T10:48:20.0875063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0875239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0875629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0875803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0876172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0876355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0876732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0876923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0877172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0877417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0877820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0878283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0878555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0878785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0878891Z ok (3.702s)
2022-12-01T10:48:20.0878910Z 
2022-12-01T10:48:20.0879181Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0879292Z Ran 1 test in 3.703s
2022-12-01T10:48:20.0879313Z 
2022-12-01T10:48:20.0879406Z OK
2022-12-01T10:48:20.0879425Z 
2022-12-01T10:48:20.0879554Z Generating XML reports...
2022-12-01T10:48:20.0880010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104616.xml
2022-12-01T10:48:20.0880371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0880552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0880938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0881129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0881149Z 
2022-12-01T10:48:20.0881256Z Running tests...
2022-12-01T10:48:20.0881522Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0881839Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0882104Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0882324Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59731
2022-12-01T10:48:20.0882580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59732
2022-12-01T10:48:20.0882964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0883141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0883521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0883708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0884075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0884248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0884625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0884799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0885048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0885298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0885701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0886097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0886331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0886558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0886663Z ok (3.647s)
2022-12-01T10:48:20.0886682Z 
2022-12-01T10:48:20.0886949Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0887047Z Ran 1 test in 3.647s
2022-12-01T10:48:20.0887084Z 
2022-12-01T10:48:20.0887203Z OK
2022-12-01T10:48:20.0887224Z 
2022-12-01T10:48:20.0887358Z Generating XML reports...
2022-12-01T10:48:20.0887821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104622.xml
2022-12-01T10:48:20.0888192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0888370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0888750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0888942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0888961Z 
2022-12-01T10:48:20.0889073Z Running tests...
2022-12-01T10:48:20.0889319Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0889647Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0889937Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0890162Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59840
2022-12-01T10:48:20.0890380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59841
2022-12-01T10:48:20.0890751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0890932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0891315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0891488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0891858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0892091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0892476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0892669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0892919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0893165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0893566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0893964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0894177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0894519Z STAGE:2022-12-01 10:46:31 59841:59841 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0894751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0895085Z STAGE:2022-12-01 10:46:31 59840:59840 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0895424Z STAGE:2022-12-01 10:46:31 59840:59840 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0895763Z STAGE:2022-12-01 10:46:31 59841:59841 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0895866Z ok (3.819s)
2022-12-01T10:48:20.0895885Z 
2022-12-01T10:48:20.0896153Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0896268Z Ran 1 test in 3.819s
2022-12-01T10:48:20.0896288Z 
2022-12-01T10:48:20.0896365Z OK
2022-12-01T10:48:20.0896384Z 
2022-12-01T10:48:20.0896504Z Generating XML reports...
2022-12-01T10:48:20.0897009Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104628.xml
2022-12-01T10:48:20.0897394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0897570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0897950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0898142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0898162Z 
2022-12-01T10:48:20.0898271Z Running tests...
2022-12-01T10:48:20.0898538Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0898833Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0899126Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0899352Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59953
2022-12-01T10:48:20.0899570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59954
2022-12-01T10:48:20.0899943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0900115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0900485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0900661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0901022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0901269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0901656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0901846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0902092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0902337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0902737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0903140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0903370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0903695Z STAGE:2022-12-01 10:46:37 59954:59954 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0903927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0904259Z STAGE:2022-12-01 10:46:37 59953:59953 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0904593Z STAGE:2022-12-01 10:46:37 59953:59953 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0904923Z STAGE:2022-12-01 10:46:37 59954:59954 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0905029Z ok (3.797s)
2022-12-01T10:48:20.0905048Z 
2022-12-01T10:48:20.0905313Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0905424Z Ran 1 test in 3.797s
2022-12-01T10:48:20.0905444Z 
2022-12-01T10:48:20.0905538Z OK
2022-12-01T10:48:20.0905557Z 
2022-12-01T10:48:20.0905663Z Generating XML reports...
2022-12-01T10:48:20.0906166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104634.xml
2022-12-01T10:48:20.0906552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0906732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0907113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0907307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0907326Z 
2022-12-01T10:48:20.0907436Z Running tests...
2022-12-01T10:48:20.0907702Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0908023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0908282Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0908509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60066
2022-12-01T10:48:20.0908730Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60067
2022-12-01T10:48:20.0909101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0909280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0909911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0910106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0910474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0910632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0911005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0911287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0911532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0911777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0912182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0912580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0912809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0913145Z STAGE:2022-12-01 10:46:43 60067:60067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0913359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0913698Z STAGE:2022-12-01 10:46:43 60066:60066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0914040Z STAGE:2022-12-01 10:46:43 60067:60067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0914370Z STAGE:2022-12-01 10:46:43 60066:60066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0914474Z ok (3.800s)
2022-12-01T10:48:20.0914494Z 
2022-12-01T10:48:20.0914761Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0914873Z Ran 1 test in 3.800s
2022-12-01T10:48:20.0914893Z 
2022-12-01T10:48:20.0914985Z OK
2022-12-01T10:48:20.0915004Z 
2022-12-01T10:48:20.0915131Z Generating XML reports...
2022-12-01T10:48:20.0915569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104640.xml
2022-12-01T10:48:20.0916002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0916189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0916573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0916764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0916783Z 
2022-12-01T10:48:20.0916891Z Running tests...
2022-12-01T10:48:20.0917156Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0917472Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0917690Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:48:20.0917728Z 
2022-12-01T10:48:20.0917969Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0918084Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0918104Z 
2022-12-01T10:48:20.0918217Z OK (skipped=1)
2022-12-01T10:48:20.0918236Z 
2022-12-01T10:48:20.0918361Z Generating XML reports...
2022-12-01T10:48:20.0918812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104645.xml
2022-12-01T10:48:20.0919184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0919363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0919748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0919921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0919958Z 
2022-12-01T10:48:20.0920049Z Running tests...
2022-12-01T10:48:20.0920314Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0920703Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0920969Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:48:20.0920989Z 
2022-12-01T10:48:20.0921254Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0921369Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0921389Z 
2022-12-01T10:48:20.0921497Z OK (skipped=1)
2022-12-01T10:48:20.0921515Z 
2022-12-01T10:48:20.0921641Z Generating XML reports...
2022-12-01T10:48:20.0922077Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104647.xml
2022-12-01T10:48:20.0922452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0922630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0923016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0923208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0923227Z 
2022-12-01T10:48:20.0923335Z Running tests...
2022-12-01T10:48:20.0923602Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0923916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0924177Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T10:48:20.0924196Z 
2022-12-01T10:48:20.0924439Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0924554Z Ran 1 test in 0.002s
2022-12-01T10:48:20.0924574Z 
2022-12-01T10:48:20.0924681Z OK (skipped=1)
2022-12-01T10:48:20.0924700Z 
2022-12-01T10:48:20.0924825Z Generating XML reports...
2022-12-01T10:48:20.0925323Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104649.xml
2022-12-01T10:48:20.0925712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0925890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0926273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0926463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0926483Z 
2022-12-01T10:48:20.0926571Z Running tests...
2022-12-01T10:48:20.0926835Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0927147Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0927420Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0927650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60278
2022-12-01T10:48:20.0927869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60279
2022-12-01T10:48:20.0928241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0928416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0928794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0928969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0929334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0929506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0929952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0930139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0930388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0930633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0931039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0931417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0931649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0931987Z STAGE:2022-12-01 10:46:55 60279:60279 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0932224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0932558Z STAGE:2022-12-01 10:46:55 60278:60278 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0932900Z STAGE:2022-12-01 10:46:55 60279:60279 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0933230Z STAGE:2022-12-01 10:46:55 60278:60278 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0933332Z ok (3.835s)
2022-12-01T10:48:20.0933352Z 
2022-12-01T10:48:20.0933619Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0933714Z Ran 1 test in 3.835s
2022-12-01T10:48:20.0933749Z 
2022-12-01T10:48:20.0933824Z OK
2022-12-01T10:48:20.0933843Z 
2022-12-01T10:48:20.0933968Z Generating XML reports...
2022-12-01T10:48:20.0934420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104651.xml
2022-12-01T10:48:20.0934845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0935028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0935414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0935604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0935624Z 
2022-12-01T10:48:20.0935733Z Running tests...
2022-12-01T10:48:20.0935977Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0936291Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0936550Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0936771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60391
2022-12-01T10:48:20.0936997Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60392
2022-12-01T10:48:20.0937370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0937544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0937927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0938100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0938465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0938642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0939019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0939302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0939553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0939799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0940201Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0940598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0940810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0941038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0941143Z ok (3.788s)
2022-12-01T10:48:20.0941162Z 
2022-12-01T10:48:20.0941430Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0941549Z Ran 1 test in 3.788s
2022-12-01T10:48:20.0941571Z 
2022-12-01T10:48:20.0941665Z OK
2022-12-01T10:48:20.0941685Z 
2022-12-01T10:48:20.0941811Z Generating XML reports...
2022-12-01T10:48:20.0942259Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104657.xml
2022-12-01T10:48:20.0942633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0942791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0943172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0943368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0943388Z 
2022-12-01T10:48:20.0943495Z Running tests...
2022-12-01T10:48:20.0943763Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0944148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0944440Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0944660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60500
2022-12-01T10:48:20.0944858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60501
2022-12-01T10:48:20.0945231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0945411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0945793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0945982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0946357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0946532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0946910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0947096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0947325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0947570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0947970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0948364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0948651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0948990Z STAGE:2022-12-01 10:47:06 60501:60501 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0949219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0949779Z STAGE:2022-12-01 10:47:06 60500:60500 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0950135Z STAGE:2022-12-01 10:47:06 60500:60500 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0950449Z STAGE:2022-12-01 10:47:06 60501:60501 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0950553Z ok (3.708s)
2022-12-01T10:48:20.0950573Z 
2022-12-01T10:48:20.0950835Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0950945Z Ran 1 test in 3.708s
2022-12-01T10:48:20.0950970Z 
2022-12-01T10:48:20.0951062Z OK
2022-12-01T10:48:20.0951081Z 
2022-12-01T10:48:20.0951212Z Generating XML reports...
2022-12-01T10:48:20.0951667Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104703.xml
2022-12-01T10:48:20.0952041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0952216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0952580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0952773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0952793Z 
2022-12-01T10:48:20.0952899Z Running tests...
2022-12-01T10:48:20.0953160Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0953477Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0953834Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0954067Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60613
2022-12-01T10:48:20.0954286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60614
2022-12-01T10:48:20.0954644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0954823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0955190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0955365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0955745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0955941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0956323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0956514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0956762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0956989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0957388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0957786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0958021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0958322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0958666Z STAGE:2022-12-01 10:47:12 60614:60614 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0958991Z STAGE:2022-12-01 10:47:12 60613:60613 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T10:48:20.0959327Z STAGE:2022-12-01 10:47:12 60613:60613 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0959660Z STAGE:2022-12-01 10:47:12 60614:60614 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T10:48:20.0959745Z ok (3.719s)
2022-12-01T10:48:20.0959764Z 
2022-12-01T10:48:20.0960027Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0960141Z Ran 1 test in 3.719s
2022-12-01T10:48:20.0960160Z 
2022-12-01T10:48:20.0960253Z OK
2022-12-01T10:48:20.0960272Z 
2022-12-01T10:48:20.0960399Z Generating XML reports...
2022-12-01T10:48:20.0960859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104709.xml
2022-12-01T10:48:20.0961236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0961412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0961793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0961964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0961984Z 
2022-12-01T10:48:20.0962093Z Running tests...
2022-12-01T10:48:20.0962357Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0962669Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0962933Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0963204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60726
2022-12-01T10:48:20.0963428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60727
2022-12-01T10:48:20.0963803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0963960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0964341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0964535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0964897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0965070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0965452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0965644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0965892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0966141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0966523Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0966923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0967157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0967387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0967541Z ok (3.813s)
2022-12-01T10:48:20.0967561Z 
2022-12-01T10:48:20.0967834Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0967949Z Ran 1 test in 3.813s
2022-12-01T10:48:20.0967968Z 
2022-12-01T10:48:20.0968063Z OK
2022-12-01T10:48:20.0968082Z 
2022-12-01T10:48:20.0968208Z Generating XML reports...
2022-12-01T10:48:20.0968642Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104714.xml
2022-12-01T10:48:20.0969017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0969196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0969574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0969768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0969791Z 
2022-12-01T10:48:20.0969910Z Running tests...
2022-12-01T10:48:20.0970175Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0970490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0970742Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0970963Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60910
2022-12-01T10:48:20.0971183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60911
2022-12-01T10:48:20.0971557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0971736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0972112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0972309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0972718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0972898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0973261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0973451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0973695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0973939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0974342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0974746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0974975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0975206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0975306Z ok (5.207s)
2022-12-01T10:48:20.0975326Z 
2022-12-01T10:48:20.0975572Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0975686Z Ran 1 test in 5.207s
2022-12-01T10:48:20.0975705Z 
2022-12-01T10:48:20.0975798Z OK
2022-12-01T10:48:20.0975817Z 
2022-12-01T10:48:20.0975941Z Generating XML reports...
2022-12-01T10:48:20.0976397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104720.xml
2022-12-01T10:48:20.0976764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0976992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0977388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0977562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0977602Z 
2022-12-01T10:48:20.0977693Z Running tests...
2022-12-01T10:48:20.0977959Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0978275Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0978593Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0978813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61111
2022-12-01T10:48:20.0979029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61112
2022-12-01T10:48:20.0979411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0979589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0979950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0980144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0980512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0980691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0981067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0981257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0981504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0981804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0982200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0982593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0982826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0983057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0983157Z ok (5.662s)
2022-12-01T10:48:20.0983176Z 
2022-12-01T10:48:20.0983443Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0983557Z Ran 1 test in 5.662s
2022-12-01T10:48:20.0983576Z 
2022-12-01T10:48:20.0983673Z OK
2022-12-01T10:48:20.0983692Z 
2022-12-01T10:48:20.0983817Z Generating XML reports...
2022-12-01T10:48:20.0984250Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104728.xml
2022-12-01T10:48:20.0984623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0984801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0985181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0985370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0985390Z 
2022-12-01T10:48:20.0985500Z Running tests...
2022-12-01T10:48:20.0985765Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0986082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0986417Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0986619Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61226
2022-12-01T10:48:20.0986840Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61227
2022-12-01T10:48:20.0987212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0987392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0987771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0987963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0988324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0988504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0988870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0989063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0989305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0989781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0990200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0990598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0990827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0991058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.0991391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8i3tygl
2022-12-01T10:48:20.0991640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqkifger6
2022-12-01T10:48:20.0991912Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8i3tygl/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0992182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqkifger6/_remote_module_non_scriptable.py
2022-12-01T10:48:20.0992285Z ok (3.800s)
2022-12-01T10:48:20.0992305Z 
2022-12-01T10:48:20.0992575Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0992690Z Ran 1 test in 3.800s
2022-12-01T10:48:20.0992709Z 
2022-12-01T10:48:20.0992802Z OK
2022-12-01T10:48:20.0992821Z 
2022-12-01T10:48:20.0992943Z Generating XML reports...
2022-12-01T10:48:20.0993397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104735.xml
2022-12-01T10:48:20.0993759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0993940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0994323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0994516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0994536Z 
2022-12-01T10:48:20.0994649Z Running tests...
2022-12-01T10:48:20.0994913Z ----------------------------------------------------------------------
2022-12-01T10:48:20.0995231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.0995483Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.0995755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61339
2022-12-01T10:48:20.0995976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61340
2022-12-01T10:48:20.0996354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0996531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0996912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0997101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0997469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.0997647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.0998019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.0998196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.0998445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.0998692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.0999101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0999498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.0999735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.0999963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.1000219Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfl8g4_mr
2022-12-01T10:48:20.1000540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfl8g4_mr/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1000786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96al7u5n
2022-12-01T10:48:20.1001051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96al7u5n/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1001161Z ok (5.169s)
2022-12-01T10:48:20.1001181Z 
2022-12-01T10:48:20.1001452Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1001571Z Ran 1 test in 5.169s
2022-12-01T10:48:20.1001590Z 
2022-12-01T10:48:20.1001682Z OK
2022-12-01T10:48:20.1001701Z 
2022-12-01T10:48:20.1001828Z Generating XML reports...
2022-12-01T10:48:20.1002284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104741.xml
2022-12-01T10:48:20.1002656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1002822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1003213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1003405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1003425Z 
2022-12-01T10:48:20.1003540Z Running tests...
2022-12-01T10:48:20.1003802Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1004115Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.1004412Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.1004634Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61450
2022-12-01T10:48:20.1004834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61451
2022-12-01T10:48:20.1005270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1005445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1005826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1006020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1006382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1025503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1026011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1026217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1026522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.1026771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.1027166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.1027635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.1027873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.1028110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.1028367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_aof6ntj
2022-12-01T10:48:20.1028647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_aof6ntj/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1029021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgdav95z
2022-12-01T10:48:20.1029307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgdav95z/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1030463Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:20.1031254Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T10:48:20.1031373Z ok (5.607s)
2022-12-01T10:48:20.1031396Z 
2022-12-01T10:48:20.1031685Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1031783Z Ran 1 test in 5.607s
2022-12-01T10:48:20.1031821Z 
2022-12-01T10:48:20.1031896Z OK
2022-12-01T10:48:20.1031915Z 
2022-12-01T10:48:20.1032040Z Generating XML reports...
2022-12-01T10:48:20.1032507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104748.xml
2022-12-01T10:48:20.1032889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1033186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1033576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1033776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1033797Z 
2022-12-01T10:48:20.1033907Z Running tests...
2022-12-01T10:48:20.1034154Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1034472Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.1034763Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.1034985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61565
2022-12-01T10:48:20.1035210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61566
2022-12-01T10:48:20.1035596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1035780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1036164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1036358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1036710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1036884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1037266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1037455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1037706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.1038030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.1038452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.1038851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.1039065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.1039296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.1039533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.1039777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.1040185Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.1040578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.1040819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:20.1041056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:20.1041453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.1041848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.1042094Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksobbeib
2022-12-01T10:48:20.1042432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksobbeib/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1042689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsasvhoh2
2022-12-01T10:48:20.1042958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsasvhoh2/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1043063Z ok (10.399s)
2022-12-01T10:48:20.1043084Z 
2022-12-01T10:48:20.1043354Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1043469Z Ran 1 test in 10.399s
2022-12-01T10:48:20.1043488Z 
2022-12-01T10:48:20.1043581Z OK
2022-12-01T10:48:20.1043601Z 
2022-12-01T10:48:20.1043707Z Generating XML reports...
2022-12-01T10:48:20.1044164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104756.xml
2022-12-01T10:48:20.1044540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1044720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1045106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1045297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1045316Z 
2022-12-01T10:48:20.1045427Z Running tests...
2022-12-01T10:48:20.1045693Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1046009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T10:48:20.1046280Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T10:48:20.1046500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61688
2022-12-01T10:48:20.1046719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61689
2022-12-01T10:48:20.1047143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1047329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1047711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1047902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1048263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T10:48:20.1048418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T10:48:20.1048792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T10:48:20.1048981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T10:48:20.1049229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T10:48:20.1049482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T10:48:20.1049882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.1050279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T10:48:20.1050511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T10:48:20.1050739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T10:48:20.1050961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T10:48:20.1051203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T10:48:20.1051666Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.1052060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T10:48:20.1052301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T10:48:20.1052540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T10:48:20.1052935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.1053327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T10:48:20.1053585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8wiuoun
2022-12-01T10:48:20.1053841Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8wiuoun/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1054103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfky9zirn
2022-12-01T10:48:20.1054374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfky9zirn/_remote_module_non_scriptable.py
2022-12-01T10:48:20.1054476Z ok (10.321s)
2022-12-01T10:48:20.1054496Z 
2022-12-01T10:48:20.1054765Z ----------------------------------------------------------------------
2022-12-01T10:48:20.1054882Z Ran 1 test in 10.321s
2022-12-01T10:48:20.1054902Z 
2022-12-01T10:48:20.1054995Z OK
2022-12-01T10:48:20.1055015Z 
2022-12-01T10:48:20.1055139Z Generating XML reports...
2022-12-01T10:48:20.1055596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104808.xml
2022-12-01T10:48:20.1055616Z 
2022-12-01T10:48:20.1056020Z ##[endgroup]
2022-12-01T10:48:20.1056490Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_z7t7n6hj)
2022-12-01T10:48:20.1056567Z 
2022-12-01T10:48:20.1056765Z Running distributed tests for the test backend with file init_method
2022-12-01T10:48:20.1056959Z Running distributed tests for the nccl backend with file init_method
2022-12-01T10:48:20.1057145Z Running distributed tests for the gloo backend with file init_method
2022-12-01T10:48:20.1057662Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:48:19.826572]
2022-12-01T10:48:20.1057854Z Running distributed tests for the ucc backend with file init_method
2022-12-01T10:48:20.1058369Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:48:19.828269]
2022-12-01T10:48:20.1058865Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:48:19.829168]
2022-12-01T10:48:20.1059380Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 10:48:19.830698]
2022-12-01T10:48:21.8574101Z 
2022-12-01T10:48:21.8574849Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T10:48:21.8575797Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_a7j4hb2b)
2022-12-01T10:48:21.8576288Z <unittest.suite.TestSuite tests=[]>
2022-12-01T10:48:21.8576494Z 
2022-12-01T10:48:21.8576810Z ##[endgroup]
2022-12-01T10:48:21.8577520Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_a7j4hb2b)
2022-12-01T10:48:21.8578205Z 
2022-12-01T11:08:36.2315869Z 
2022-12-01T11:08:36.2316440Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T11:08:36.2320130Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_xjn_sjyx)
2022-12-01T11:08:36.2324319Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:08:36.2363431Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T11:08:36.2404979Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2407280Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2407865Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2408446Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2408917Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2409574Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2410230Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2411272Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2411790Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2412538Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2413289Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2413993Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2414708Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2415277Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2415991Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2416481Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2417126Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2417707Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2418188Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2418791Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2419406Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2420012Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2420598Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2421165Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2421621Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2422125Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2422767Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2423397Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2423974Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2424357Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2424978Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2425398Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2426006Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2426478Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2427071Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2427657Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2428112Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2428725Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2429312Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2430338Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2431014Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2431479Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2432101Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2432533Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2433146Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2433587Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2434357Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2434888Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2435422Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2436038Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2436480Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2437064Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2437490Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2438073Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2438585Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2439107Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2439704Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2440115Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2440688Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2441095Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2441599Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2442054Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2442443Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2442982Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2443381Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2443856Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2444393Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2444911Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2445437Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2445985Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2446393Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2446956Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2447406Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2447950Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2448328Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2448936Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2449427Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2449872Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2450462Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2450929Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2451514Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2452075Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2452497Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2453148Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2453758Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2454240Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2454911Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2455466Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2456044Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2456799Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2457456Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2457901Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2458352Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2458808Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2459247Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2459721Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2460186Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2460642Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2461065Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2461473Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2461862Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2462216Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2462593Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2462987Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2463394Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2463768Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2464155Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2464565Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2464962Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2465374Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2465857Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2466263Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2466698Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2467124Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2467548Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2467955Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2468373Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2468809Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2469224Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2470333Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2470737Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2471123Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2471499Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2471892Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2472285Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2472662Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2473121Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2473634Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2474094Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2474491Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2474914Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2475478Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2475911Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2476381Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2476828Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2477237Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2477681Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2478092Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2478467Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2478842Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2479261Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2479687Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2480103Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2480547Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2480966Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2481384Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2481829Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2482334Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2482901Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2483578Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2484208Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2484812Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2485420Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2486025Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2486613Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2487218Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2487774Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2488277Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2488709Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2489104Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2489502Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2489893Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2490293Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2490718Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2491145Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2491679Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2492153Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2492580Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2492954Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2493373Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2493815Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2494230Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2494652Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2495073Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2495504Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2495925Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2496356Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2496767Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2497161Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2497585Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2497991Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2498397Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2498862Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2499292Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2499680Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2500061Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2500551Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2500979Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2501345Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2501711Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2502085Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2502447Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2502830Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2503204Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2503598Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2503968Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2504336Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2504701Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2505066Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2505467Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2505866Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2506224Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2506573Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2506954Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2507360Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2507758Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2508215Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2508665Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2509128Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2510157Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2510604Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2511016Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2511443Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2511866Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2512288Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2512683Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2513095Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2513491Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2513872Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2514352Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2514847Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2515322Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2515753Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2516214Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2516673Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2517088Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2517515Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2517948Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2518395Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2518920Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2519417Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2519935Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2520402Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2520829Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2521229Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2521636Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2522025Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2522414Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2522802Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2523178Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2523568Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2523942Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2524292Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2524672Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2525053Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2525432Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2525842Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2526223Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2526596Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2526967Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2527450Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2527819Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2528171Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2528557Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2528931Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2529299Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2529704Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2530090Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2530479Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2530832Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2531209Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2531636Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2532071Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2532508Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2532907Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2533296Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2533726Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2534141Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2534538Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2534938Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2535382Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2535806Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2536255Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2536678Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2537083Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2537469Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2537875Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2538328Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2538775Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2539507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2539970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2540563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2541041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2541258Z 
2022-12-01T11:08:36.2541368Z Running tests...
2022-12-01T11:08:36.2541783Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2542319Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2542891Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2543456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62022
2022-12-01T11:08:36.2543904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62023
2022-12-01T11:08:36.2544521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2545029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2545617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2546091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2546658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2547103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2547682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2548148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2548588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2549084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2550501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2551205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2551714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2552191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2552718Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2553559Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2554209Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2555142Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2555772Z [1669891709.759587] [7ff5451e42d4:62022:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2556254Z [1669891709.759587] [7ff5451e42d4:62022:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2556712Z [1669891709.765199] [7ff5451e42d4:62023:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2557187Z [1669891709.765199] [7ff5451e42d4:62023:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2557713Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2558563Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2559208Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2560032Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2560687Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2561509Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2562157Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2563086Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2563738Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2564553Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2565196Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:08:36.2566016Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:08:36.2566499Z ok (7.275s)
2022-12-01T11:08:36.2566651Z 
2022-12-01T11:08:36.2566928Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2567245Z Ran 1 test in 7.275s
2022-12-01T11:08:36.2567408Z 
2022-12-01T11:08:36.2567501Z OK
2022-12-01T11:08:36.2567637Z 
2022-12-01T11:08:36.2567763Z Generating XML reports...
2022-12-01T11:08:36.2568356Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104823.xml
2022-12-01T11:08:36.2569075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2569530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2570112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2570620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2570852Z 
2022-12-01T11:08:36.2570962Z Running tests...
2022-12-01T11:08:36.2571376Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2571951Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2572492Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-12-01T11:08:36.2572804Z 
2022-12-01T11:08:36.2573075Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2573404Z Ran 1 test in 0.004s
2022-12-01T11:08:36.2573566Z 
2022-12-01T11:08:36.2573657Z OK (skipped=1)
2022-12-01T11:08:36.2573811Z 
2022-12-01T11:08:36.2573937Z Generating XML reports...
2022-12-01T11:08:36.2574543Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104833.xml
2022-12-01T11:08:36.2575261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2575696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2576283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2576759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2576990Z 
2022-12-01T11:08:36.2577080Z Running tests...
2022-12-01T11:08:36.2577489Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2578022Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2578530Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2579005Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62466
2022-12-01T11:08:36.2579454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62467
2022-12-01T11:08:36.2580067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2580582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2581170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2581642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2582223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2582649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2583227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2583696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2584136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2584645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2585312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2586006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2586510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2586981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2587329Z ok (4.149s)
2022-12-01T11:08:36.2587479Z 
2022-12-01T11:08:36.2587755Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2588066Z Ran 1 test in 4.149s
2022-12-01T11:08:36.2588228Z 
2022-12-01T11:08:36.2588324Z OK
2022-12-01T11:08:36.2588458Z 
2022-12-01T11:08:36.2588589Z Generating XML reports...
2022-12-01T11:08:36.2589238Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104835.xml
2022-12-01T11:08:36.2590273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2590725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2591305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2591762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2591993Z 
2022-12-01T11:08:36.2592103Z Running tests...
2022-12-01T11:08:36.2592510Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2593027Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2593558Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2594637Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2022-12-01T11:08:36.2595162Z 
2022-12-01T11:08:36.2595431Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2595762Z Ran 1 test in 1.615s
2022-12-01T11:08:36.2595904Z 
2022-12-01T11:08:36.2596014Z OK (skipped=1)
2022-12-01T11:08:36.2596169Z 
2022-12-01T11:08:36.2596295Z Generating XML reports...
2022-12-01T11:08:36.2596902Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104841.xml
2022-12-01T11:08:36.2597599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2598170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2598754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2599228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2599459Z 
2022-12-01T11:08:36.2599552Z Running tests...
2022-12-01T11:08:36.2599961Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2600496Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2601017Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2601536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62914
2022-12-01T11:08:36.2601988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62915
2022-12-01T11:08:36.2602611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2603044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2603628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2604101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2604689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2605123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2605705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2606170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2606605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2607186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2607865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2608560Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2609065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2609570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvwke3y_s
2022-12-01T11:08:36.2610120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvwke3y_s/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2610634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2611122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeln7xaey
2022-12-01T11:08:36.2611663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeln7xaey/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2612179Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2612643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2613131Z [1669891730.394738] [7ff5451e42d4:62915:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2613613Z [1669891730.394738] [7ff5451e42d4:62915:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2614088Z [1669891730.398247] [7ff5451e42d4:62914:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2614545Z [1669891730.398247] [7ff5451e42d4:62914:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2614959Z ok (5.989s)
2022-12-01T11:08:36.2615107Z 
2022-12-01T11:08:36.2615385Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2615718Z Ran 1 test in 5.990s
2022-12-01T11:08:36.2615860Z 
2022-12-01T11:08:36.2615954Z OK
2022-12-01T11:08:36.2616090Z 
2022-12-01T11:08:36.2616217Z Generating XML reports...
2022-12-01T11:08:36.2616824Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104845.xml
2022-12-01T11:08:36.2617521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2617975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2618555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2619026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2619260Z 
2022-12-01T11:08:36.2619375Z Running tests...
2022-12-01T11:08:36.2619783Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2620293Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2620847Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2621378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63327
2022-12-01T11:08:36.2621828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63328
2022-12-01T11:08:36.2622419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2622867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2623446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2624052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2624650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2625094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2625670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2626118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2626573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2627073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2627713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2628408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2628931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2629788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ziw2y6b
2022-12-01T11:08:36.2630322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ziw2y6b/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2630829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2631322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuor10rgq
2022-12-01T11:08:36.2631854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuor10rgq/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2632351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2632949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2633433Z [1669891738.648988] [7ff5451e42d4:63328:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2633894Z [1669891738.648988] [7ff5451e42d4:63328:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2634365Z [1669891738.627213] [7ff5451e42d4:63327:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2634838Z [1669891738.627213] [7ff5451e42d4:63327:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2635180Z ok (5.985s)
2022-12-01T11:08:36.2635331Z 
2022-12-01T11:08:36.2635602Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2635942Z Ran 1 test in 5.986s
2022-12-01T11:08:36.2636101Z 
2022-12-01T11:08:36.2636195Z OK
2022-12-01T11:08:36.2636331Z 
2022-12-01T11:08:36.2636456Z Generating XML reports...
2022-12-01T11:08:36.2637046Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104853.xml
2022-12-01T11:08:36.2637764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2638216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2638776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2639247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2639476Z 
2022-12-01T11:08:36.2639584Z Running tests...
2022-12-01T11:08:36.2639989Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2640507Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2641128Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2641675Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63668
2022-12-01T11:08:36.2642104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63669
2022-12-01T11:08:36.2642719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2643167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2643741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2644194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2644778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2645232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2645814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2646262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2646715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2647209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2647848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2648543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2649064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2649603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2650084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvcx2t6jt
2022-12-01T11:08:36.2650625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvcx2t6jt/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2651157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnsbeqre
2022-12-01T11:08:36.2651689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnsbeqre/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2652179Z [1669891746.962820] [7ff5451e42d4:63668:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2652659Z [1669891746.962820] [7ff5451e42d4:63668:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2653140Z [1669891746.968135] [7ff5451e42d4:63669:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2653614Z [1669891746.968135] [7ff5451e42d4:63669:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2654072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2654559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2655037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2655501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2655978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2656449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2656930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2657438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2657921Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2658392Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2658848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2659319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2659665Z ok (7.290s)
2022-12-01T11:08:36.2659813Z 
2022-12-01T11:08:36.2660093Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2660409Z Ran 1 test in 7.290s
2022-12-01T11:08:36.2660570Z 
2022-12-01T11:08:36.2660659Z OK
2022-12-01T11:08:36.2660792Z 
2022-12-01T11:08:36.2660924Z Generating XML reports...
2022-12-01T11:08:36.2661520Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104901.xml
2022-12-01T11:08:36.2662238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2662687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2663265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2663719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2663948Z 
2022-12-01T11:08:36.2664058Z Running tests...
2022-12-01T11:08:36.2664464Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2664975Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2665541Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2666144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64024
2022-12-01T11:08:36.2666596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64025
2022-12-01T11:08:36.2667193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2667640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2668217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2668688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2669254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2670000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2670627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2671077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2671532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2672026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2672681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2673355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2673876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2674348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2674914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4y01qzy
2022-12-01T11:08:36.2675469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4y01qzy/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2675998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4se_ux3
2022-12-01T11:08:36.2676530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4se_ux3/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2677024Z [1669891756.662211] [7ff5451e42d4:64025:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2677500Z [1669891756.662211] [7ff5451e42d4:64025:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2677969Z [1669891756.662308] [7ff5451e42d4:64024:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2678451Z [1669891756.662308] [7ff5451e42d4:64024:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2678910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2679398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2679876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2680352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2680683Z ok (6.253s)
2022-12-01T11:08:36.2680863Z 
2022-12-01T11:08:36.2681148Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2681482Z Ran 1 test in 6.254s
2022-12-01T11:08:36.2681642Z 
2022-12-01T11:08:36.2681719Z OK
2022-12-01T11:08:36.2681852Z 
2022-12-01T11:08:36.2681976Z Generating XML reports...
2022-12-01T11:08:36.2682701Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104911.xml
2022-12-01T11:08:36.2683418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2683856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2684432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2684897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2685124Z 
2022-12-01T11:08:36.2685214Z Running tests...
2022-12-01T11:08:36.2685617Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2686149Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2686720Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2687259Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64377
2022-12-01T11:08:36.2687709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64378
2022-12-01T11:08:36.2688320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2688753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2689333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2689802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2690378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2690802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2691434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2691913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2692367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2692846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2693505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2694198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2694699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2695168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2695665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7f_qwmq1
2022-12-01T11:08:36.2696205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7f_qwmq1/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2696714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_1kjbrw
2022-12-01T11:08:36.2697239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_1kjbrw/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2697745Z [1669891765.098336] [7ff5451e42d4:64378:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2698224Z [1669891765.098336] [7ff5451e42d4:64378:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2698682Z [1669891765.094395] [7ff5451e42d4:64377:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2699220Z [1669891765.094395] [7ff5451e42d4:64377:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2699697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2700187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2700649Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2701128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2701475Z ok (6.246s)
2022-12-01T11:08:36.2701623Z 
2022-12-01T11:08:36.2701883Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2702213Z Ran 1 test in 6.247s
2022-12-01T11:08:36.2702371Z 
2022-12-01T11:08:36.2702466Z OK
2022-12-01T11:08:36.2702600Z 
2022-12-01T11:08:36.2702707Z Generating XML reports...
2022-12-01T11:08:36.2703326Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104919.xml
2022-12-01T11:08:36.2704045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2704493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2705051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2705519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2705747Z 
2022-12-01T11:08:36.2705856Z Running tests...
2022-12-01T11:08:36.2706268Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2706781Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2707368Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2707987Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64721
2022-12-01T11:08:36.2708427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64722
2022-12-01T11:08:36.2709037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2709847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2710441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2710893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2711476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2711920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2712489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2712951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2713406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2713900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2714542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2715233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2715754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2716223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2716805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp64nt_fag
2022-12-01T11:08:36.2717340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp64nt_fag/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2717871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94rdx08f
2022-12-01T11:08:36.2718388Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94rdx08f/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2718901Z [1669891773.424201] [7ff5451e42d4:64721:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2719376Z [1669891773.424201] [7ff5451e42d4:64721:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2719850Z [1669891773.431137] [7ff5451e42d4:64722:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2720310Z [1669891773.431137] [7ff5451e42d4:64722:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2720786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2721274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2721626Z ok (6.355s)
2022-12-01T11:08:36.2721758Z 
2022-12-01T11:08:36.2722044Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2722376Z Ran 1 test in 6.355s
2022-12-01T11:08:36.2722537Z 
2022-12-01T11:08:36.2722630Z OK
2022-12-01T11:08:36.2722764Z 
2022-12-01T11:08:36.2722869Z Generating XML reports...
2022-12-01T11:08:36.2723471Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104928.xml
2022-12-01T11:08:36.2724189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2724652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2725287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2725768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2725998Z 
2022-12-01T11:08:36.2726107Z Running tests...
2022-12-01T11:08:36.2726492Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2727022Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2727608Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2728166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65008
2022-12-01T11:08:36.2728596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65009
2022-12-01T11:08:36.2729204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2729660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2730231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2730683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2731265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2731711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2732269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2732738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2733259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2733758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2734407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2735102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2735624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2736097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2736582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjnfa2ei
2022-12-01T11:08:36.2737122Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjnfa2ei/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2737664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtj62bq2
2022-12-01T11:08:36.2738183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtj62bq2/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2738697Z [1669891782.272002] [7ff5451e42d4:65009:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2739176Z [1669891782.272002] [7ff5451e42d4:65009:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2739646Z [1669891782.268995] [7ff5451e42d4:65008:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2740095Z [1669891782.268995] [7ff5451e42d4:65008:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2740570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2741060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2741592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2742062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2742406Z ok (6.975s)
2022-12-01T11:08:36.2742554Z 
2022-12-01T11:08:36.2742831Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2743142Z Ran 1 test in 6.975s
2022-12-01T11:08:36.2743301Z 
2022-12-01T11:08:36.2743393Z OK
2022-12-01T11:08:36.2743527Z 
2022-12-01T11:08:36.2743650Z Generating XML reports...
2022-12-01T11:08:36.2744254Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104937.xml
2022-12-01T11:08:36.2744957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2745412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2745992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2746447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2746680Z 
2022-12-01T11:08:36.2746787Z Running tests...
2022-12-01T11:08:36.2747195Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2747727Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2748269Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2748813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65391
2022-12-01T11:08:36.2749263Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65392
2022-12-01T11:08:36.2750247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2750687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2751265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2751737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2752302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2752747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2753320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2753785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2754222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2754721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2755381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2756051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2756577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2757044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2757549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb2s4efh6
2022-12-01T11:08:36.2758071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb2s4efh6/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2758607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw87eq07m
2022-12-01T11:08:36.2759222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw87eq07m/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2759749Z [1669891791.578776] [7ff5451e42d4:65391:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2760208Z [1669891791.578776] [7ff5451e42d4:65391:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2760678Z [1669891791.586374] [7ff5451e42d4:65392:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2761146Z [1669891791.586374] [7ff5451e42d4:65392:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2761617Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2762087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2762577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2763061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2763389Z ok (6.562s)
2022-12-01T11:08:36.2763539Z 
2022-12-01T11:08:36.2763820Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2764154Z Ran 1 test in 6.562s
2022-12-01T11:08:36.2764314Z 
2022-12-01T11:08:36.2764407Z OK
2022-12-01T11:08:36.2764521Z 
2022-12-01T11:08:36.2764646Z Generating XML reports...
2022-12-01T11:08:36.2765255Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104946.xml
2022-12-01T11:08:36.2765973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2766406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2767095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2767563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2767793Z 
2022-12-01T11:08:36.2767904Z Running tests...
2022-12-01T11:08:36.2768295Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2768825Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2769406Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2769950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65744
2022-12-01T11:08:36.2770448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65745
2022-12-01T11:08:36.2771061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2771518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2772077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2772547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2773126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2773569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2774122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2774588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2775040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2775581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2776262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2776952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2777471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2777924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2778419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdp7mxyqh
2022-12-01T11:08:36.2811693Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdp7mxyqh/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2812250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmperux6dvk
2022-12-01T11:08:36.2812821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmperux6dvk/_remote_module_non_scriptable.py
2022-12-01T11:08:36.2813342Z [1669891800.375556] [7ff5451e42d4:65744:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2813806Z [1669891800.375556] [7ff5451e42d4:65744:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2814282Z [1669891800.383252] [7ff5451e42d4:65745:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2814753Z [1669891800.383252] [7ff5451e42d4:65745:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2815233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2815703Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2816385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2816873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.2817229Z ok (5.997s)
2022-12-01T11:08:36.2817361Z 
2022-12-01T11:08:36.2817689Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2818025Z Ran 1 test in 5.997s
2022-12-01T11:08:36.2818184Z 
2022-12-01T11:08:36.2818279Z OK
2022-12-01T11:08:36.2818412Z 
2022-12-01T11:08:36.2818518Z Generating XML reports...
2022-12-01T11:08:36.2819133Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104955.xml
2022-12-01T11:08:36.2819858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2820311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2820886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2821365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2821596Z 
2022-12-01T11:08:36.2821706Z Running tests...
2022-12-01T11:08:36.2822104Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2822631Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2823186Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2824263Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2022-12-01T11:08:36.2824794Z 
2022-12-01T11:08:36.2825147Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2825476Z Ran 1 test in 1.677s
2022-12-01T11:08:36.2825637Z 
2022-12-01T11:08:36.2825745Z OK (skipped=1)
2022-12-01T11:08:36.2825901Z 
2022-12-01T11:08:36.2826027Z Generating XML reports...
2022-12-01T11:08:36.2826621Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105003.xml
2022-12-01T11:08:36.2827346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2827805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2828384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2828846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2829083Z 
2022-12-01T11:08:36.2829194Z Running tests...
2022-12-01T11:08:36.2829880Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2830404Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2830960Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2831494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66181
2022-12-01T11:08:36.2831953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66182
2022-12-01T11:08:36.2832550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2833009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2833588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2834171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2834763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2835217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2835796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2836247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2836704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2837213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2837877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2838565Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2839090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2839564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2839894Z ok (4.090s)
2022-12-01T11:08:36.2840041Z 
2022-12-01T11:08:36.2840319Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2840650Z Ran 1 test in 4.090s
2022-12-01T11:08:36.2840808Z 
2022-12-01T11:08:36.2840899Z OK
2022-12-01T11:08:36.2841015Z 
2022-12-01T11:08:36.2841140Z Generating XML reports...
2022-12-01T11:08:36.2841752Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105007.xml
2022-12-01T11:08:36.2842470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2842978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2843579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2844047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2844274Z 
2022-12-01T11:08:36.2844385Z Running tests...
2022-12-01T11:08:36.2844766Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2845293Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2845857Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2846934Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.728s)
2022-12-01T11:08:36.2847457Z 
2022-12-01T11:08:36.2847705Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2848030Z Ran 1 test in 1.728s
2022-12-01T11:08:36.2848194Z 
2022-12-01T11:08:36.2848301Z OK (skipped=1)
2022-12-01T11:08:36.2848454Z 
2022-12-01T11:08:36.2848577Z Generating XML reports...
2022-12-01T11:08:36.2849164Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105013.xml
2022-12-01T11:08:36.2849875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2850329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2850889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2851432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2851664Z 
2022-12-01T11:08:36.2851771Z Running tests...
2022-12-01T11:08:36.2852179Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2852691Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2853214Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2853727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66609
2022-12-01T11:08:36.2854157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66610
2022-12-01T11:08:36.2854762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2855201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2855771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2856221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2856802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2857246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2857803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2858271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2858731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2859231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2859872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2860629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2861167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2861637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2862097Z [1669891823.082550] [7ff5451e42d4:66609:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2862576Z [1669891823.082550] [7ff5451e42d4:66609:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2863048Z [1669891823.092149] [7ff5451e42d4:66610:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2863518Z [1669891823.092149] [7ff5451e42d4:66610:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2863843Z ok (6.028s)
2022-12-01T11:08:36.2863990Z 
2022-12-01T11:08:36.2864267Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2864598Z Ran 1 test in 6.028s
2022-12-01T11:08:36.2864758Z 
2022-12-01T11:08:36.2864833Z OK
2022-12-01T11:08:36.2864966Z 
2022-12-01T11:08:36.2865089Z Generating XML reports...
2022-12-01T11:08:36.2865697Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105017.xml
2022-12-01T11:08:36.2866420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2866850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2867427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2867967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2868204Z 
2022-12-01T11:08:36.2868294Z Running tests...
2022-12-01T11:08:36.2868707Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2869242Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2870024Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T11:08:36.2870295Z 
2022-12-01T11:08:36.2870615Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2870948Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2871106Z 
2022-12-01T11:08:36.2871217Z OK (skipped=1)
2022-12-01T11:08:36.2871372Z 
2022-12-01T11:08:36.2871478Z Generating XML reports...
2022-12-01T11:08:36.2872088Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105026.xml
2022-12-01T11:08:36.2872818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2873279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2873838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2874310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2874538Z 
2022-12-01T11:08:36.2874649Z Running tests...
2022-12-01T11:08:36.2875039Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2875578Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2876037Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2876544Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T11:08:36.2876852Z 
2022-12-01T11:08:36.2877193Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2877542Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2877702Z 
2022-12-01T11:08:36.2877810Z OK (skipped=1)
2022-12-01T11:08:36.2877967Z 
2022-12-01T11:08:36.2878092Z Generating XML reports...
2022-12-01T11:08:36.2878688Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105028.xml
2022-12-01T11:08:36.2879406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2879861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2880422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2880893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2881130Z 
2022-12-01T11:08:36.2881239Z Running tests...
2022-12-01T11:08:36.2881649Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2882162Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2882647Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2883169Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T11:08:36.2883476Z 
2022-12-01T11:08:36.2883729Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2884063Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2884226Z 
2022-12-01T11:08:36.2884334Z OK (skipped=1)
2022-12-01T11:08:36.2884490Z 
2022-12-01T11:08:36.2884616Z Generating XML reports...
2022-12-01T11:08:36.2885204Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105030.xml
2022-12-01T11:08:36.2886022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2886475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2887060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2887512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2887742Z 
2022-12-01T11:08:36.2887854Z Running tests...
2022-12-01T11:08:36.2888264Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2888776Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2889275Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2889832Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T11:08:36.2890160Z 
2022-12-01T11:08:36.2890430Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2890740Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2890900Z 
2022-12-01T11:08:36.2891007Z OK (skipped=1)
2022-12-01T11:08:36.2891162Z 
2022-12-01T11:08:36.2891285Z Generating XML reports...
2022-12-01T11:08:36.2891874Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105032.xml
2022-12-01T11:08:36.2892591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2893039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2893617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2894072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2894307Z 
2022-12-01T11:08:36.2894464Z Running tests...
2022-12-01T11:08:36.2894886Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2895397Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2895874Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:08:36.2896394Z Runs _test_accumulate_gradients_no_sync using default inputs ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T11:08:36.2896694Z 
2022-12-01T11:08:36.2896959Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2897268Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2897430Z 
2022-12-01T11:08:36.2897537Z OK (skipped=1)
2022-12-01T11:08:36.2897690Z 
2022-12-01T11:08:36.2897815Z Generating XML reports...
2022-12-01T11:08:36.2898400Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105034.xml
2022-12-01T11:08:36.2899129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2899579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2900151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2900600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2900832Z 
2022-12-01T11:08:36.2900939Z Running tests...
2022-12-01T11:08:36.2901340Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2901880Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2902352Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2902896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67465
2022-12-01T11:08:36.2903350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67466
2022-12-01T11:08:36.2903947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2904398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2904974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2905447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2906005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2906455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2907026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2907483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2907936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2908436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2909097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2910048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2910582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2911167Z STAGE:2022-12-01 10:50:40 67466:67466 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2911649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2912310Z STAGE:2022-12-01 10:50:40 67465:67465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2912812Z [1669891842.149326] [7ff5451e42d4:67466:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2913295Z [1669891842.149326] [7ff5451e42d4:67466:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2913770Z [1669891842.106339] [7ff5451e42d4:67465:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2914216Z [1669891842.106339] [7ff5451e42d4:67465:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2915025Z STAGE:2022-12-01 10:50:42 67466:67466 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:50:42 67465:67465 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2915422Z 
2022-12-01T11:08:36.2915761Z STAGE:2022-12-01 10:50:42 67466:67466 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2916334Z STAGE:2022-12-01 10:50:42 67465:67465 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2916905Z STAGE:2022-12-01 10:50:42 67466:67466 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2917497Z STAGE:2022-12-01 10:50:42 67465:67465 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2917839Z ok (7.099s)
2022-12-01T11:08:36.2917993Z 
2022-12-01T11:08:36.2918266Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2918580Z Ran 1 test in 7.100s
2022-12-01T11:08:36.2918746Z 
2022-12-01T11:08:36.2918840Z OK
2022-12-01T11:08:36.2918971Z 
2022-12-01T11:08:36.2919101Z Generating XML reports...
2022-12-01T11:08:36.2919695Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105036.xml
2022-12-01T11:08:36.2920548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2921007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2921566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2922038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2922268Z 
2022-12-01T11:08:36.2922379Z Running tests...
2022-12-01T11:08:36.2922789Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2923300Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2923908Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T11:08:36.2924238Z 
2022-12-01T11:08:36.2924511Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2924837Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2924981Z 
2022-12-01T11:08:36.2925089Z OK (skipped=1)
2022-12-01T11:08:36.2925242Z 
2022-12-01T11:08:36.2925366Z Generating XML reports...
2022-12-01T11:08:36.2925973Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105046.xml
2022-12-01T11:08:36.2926672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2927129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2927707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2928176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2928391Z 
2022-12-01T11:08:36.2928507Z Running tests...
2022-12-01T11:08:36.2928964Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2929513Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2930036Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T11:08:36.2930357Z 
2022-12-01T11:08:36.2930622Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2930951Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2931115Z 
2022-12-01T11:08:36.2931222Z OK (skipped=1)
2022-12-01T11:08:36.2931375Z 
2022-12-01T11:08:36.2931482Z Generating XML reports...
2022-12-01T11:08:36.2932088Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105048.xml
2022-12-01T11:08:36.2932811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2933271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2933833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2934304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2934534Z 
2022-12-01T11:08:36.2934643Z Running tests...
2022-12-01T11:08:36.2935027Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2935570Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2936108Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T11:08:36.2936420Z 
2022-12-01T11:08:36.2936687Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2936993Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2937220Z 
2022-12-01T11:08:36.2937331Z OK (skipped=1)
2022-12-01T11:08:36.2937487Z 
2022-12-01T11:08:36.2937615Z Generating XML reports...
2022-12-01T11:08:36.2938199Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105050.xml
2022-12-01T11:08:36.2938921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2939371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2939950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2940408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2940638Z 
2022-12-01T11:08:36.2940749Z Running tests...
2022-12-01T11:08:36.2941158Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2941672Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2942220Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.002s)
2022-12-01T11:08:36.2942539Z 
2022-12-01T11:08:36.2942808Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2943137Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2943299Z 
2022-12-01T11:08:36.2943388Z OK (skipped=1)
2022-12-01T11:08:36.2943543Z 
2022-12-01T11:08:36.2943667Z Generating XML reports...
2022-12-01T11:08:36.2944268Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105052.xml
2022-12-01T11:08:36.2944983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2945143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2945528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2945775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2945796Z 
2022-12-01T11:08:36.2945913Z Running tests...
2022-12-01T11:08:36.2946192Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2946506Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2946801Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support all_gather_coalesced (0.003s)
2022-12-01T11:08:36.2946821Z 
2022-12-01T11:08:36.2947087Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2947183Z Ran 1 test in 0.003s
2022-12-01T11:08:36.2947219Z 
2022-12-01T11:08:36.2947310Z OK (skipped=1)
2022-12-01T11:08:36.2947328Z 
2022-12-01T11:08:36.2947457Z Generating XML reports...
2022-12-01T11:08:36.2947905Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105054.xml
2022-12-01T11:08:36.2948292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2948474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2948855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2949047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2949067Z 
2022-12-01T11:08:36.2949175Z Running tests...
2022-12-01T11:08:36.2949740Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2950077Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2950338Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2950682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68356
2022-12-01T11:08:36.2950908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68357
2022-12-01T11:08:36.2951291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2951470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2951851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2952048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2952397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2952572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2952953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2953149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2953398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2953644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2954050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2954459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2954671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2955015Z STAGE:2022-12-01 10:51:00 68356:68356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2955246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2955655Z STAGE:2022-12-01 10:51:00 68357:68357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2955906Z [1669891862.211837] [7ff5451e42d4:68356:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2956148Z [1669891862.211837] [7ff5451e42d4:68356:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2956381Z [1669891862.246116] [7ff5451e42d4:68357:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2956614Z [1669891862.246116] [7ff5451e42d4:68357:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2957174Z STAGE:2022-12-01 10:51:02 68356:68356 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:51:02 68357:68357 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2957201Z 
2022-12-01T11:08:36.2957545Z STAGE:2022-12-01 10:51:02 68356:68356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2957875Z STAGE:2022-12-01 10:51:02 68357:68357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2958194Z STAGE:2022-12-01 10:51:02 68356:68356 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2958522Z STAGE:2022-12-01 10:51:02 68357:68357 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2958626Z ok (7.308s)
2022-12-01T11:08:36.2958645Z 
2022-12-01T11:08:36.2958914Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2959028Z Ran 1 test in 7.309s
2022-12-01T11:08:36.2959048Z 
2022-12-01T11:08:36.2959139Z OK
2022-12-01T11:08:36.2959158Z 
2022-12-01T11:08:36.2959282Z Generating XML reports...
2022-12-01T11:08:36.2959731Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105056.xml
2022-12-01T11:08:36.2960153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2960332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2960717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2960913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2960932Z 
2022-12-01T11:08:36.2961040Z Running tests...
2022-12-01T11:08:36.2961310Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2961623Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2961885Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T11:08:36.2961909Z 
2022-12-01T11:08:36.2962175Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2962270Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2962289Z 
2022-12-01T11:08:36.2962395Z OK (skipped=1)
2022-12-01T11:08:36.2962414Z 
2022-12-01T11:08:36.2962536Z Generating XML reports...
2022-12-01T11:08:36.2962987Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105106.xml
2022-12-01T11:08:36.2963365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2963544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2963925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2964119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2964143Z 
2022-12-01T11:08:36.2964255Z Running tests...
2022-12-01T11:08:36.2964551Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2964880Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2965156Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T11:08:36.2965175Z 
2022-12-01T11:08:36.2965440Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2965555Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2965575Z 
2022-12-01T11:08:36.2965683Z OK (skipped=1)
2022-12-01T11:08:36.2965702Z 
2022-12-01T11:08:36.2965824Z Generating XML reports...
2022-12-01T11:08:36.2966268Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105108.xml
2022-12-01T11:08:36.2966625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2966812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2967194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2967386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2967406Z 
2022-12-01T11:08:36.2967516Z Running tests...
2022-12-01T11:08:36.2967781Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2968091Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2968354Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2968572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68997
2022-12-01T11:08:36.2968771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68998
2022-12-01T11:08:36.2969216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2969392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2969775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2969967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2970333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2970558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2970939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2971108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2971357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2971608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2972014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2972417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2972651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2972891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.2973120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2973361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.2973799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.2974213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.2974555Z STAGE:2022-12-01 10:51:14 68998:68998 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2974892Z STAGE:2022-12-01 10:51:14 68997:68997 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2975132Z [1669891875.777858] [7ff5451e42d4:68998:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2975377Z [1669891875.777858] [7ff5451e42d4:68998:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2975603Z [1669891875.785171] [7ff5451e42d4:68997:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.2975847Z [1669891875.785171] [7ff5451e42d4:68997:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.2976402Z STAGE:2022-12-01 10:51:16 68998:68998 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:51:16 68997:68997 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2976424Z 
2022-12-01T11:08:36.2976755Z STAGE:2022-12-01 10:51:16 68998:68998 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2977058Z STAGE:2022-12-01 10:51:16 68997:68997 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.2977402Z STAGE:2022-12-01 10:51:16 68998:68998 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2977733Z STAGE:2022-12-01 10:51:16 68997:68997 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.2977835Z ok (7.120s)
2022-12-01T11:08:36.2977854Z 
2022-12-01T11:08:36.2978182Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2978299Z Ran 1 test in 7.120s
2022-12-01T11:08:36.2978319Z 
2022-12-01T11:08:36.2978412Z OK
2022-12-01T11:08:36.2978431Z 
2022-12-01T11:08:36.2978556Z Generating XML reports...
2022-12-01T11:08:36.2979010Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105110.xml
2022-12-01T11:08:36.2979368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2979544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2979926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2980117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2980136Z 
2022-12-01T11:08:36.2980247Z Running tests...
2022-12-01T11:08:36.2980520Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2980836Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2981094Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.2981295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69425
2022-12-01T11:08:36.2981514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69426
2022-12-01T11:08:36.2981890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2982068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2982449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2982642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2983062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2983249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2983630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2983802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2984050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.2984293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.2984697Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2985098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.2985339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.2985569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.2985730Z skip: Skipped due to small world size. (3.963s)
2022-12-01T11:08:36.2985750Z 
2022-12-01T11:08:36.2986020Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2986115Z Ran 1 test in 3.963s
2022-12-01T11:08:36.2986134Z 
2022-12-01T11:08:36.2986244Z OK (skipped=1)
2022-12-01T11:08:36.2986262Z 
2022-12-01T11:08:36.2986387Z Generating XML reports...
2022-12-01T11:08:36.2986833Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105119.xml
2022-12-01T11:08:36.2987207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2987451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2987840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2988035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2988054Z 
2022-12-01T11:08:36.2988164Z Running tests...
2022-12-01T11:08:36.2988412Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2988727Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2989024Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T11:08:36.2989045Z 
2022-12-01T11:08:36.2989305Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2989670Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2989691Z 
2022-12-01T11:08:36.2989814Z OK (skipped=1)
2022-12-01T11:08:36.2989833Z 
2022-12-01T11:08:36.2989961Z Generating XML reports...
2022-12-01T11:08:36.2990428Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105125.xml
2022-12-01T11:08:36.2990784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2990961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2991343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2991537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2991556Z 
2022-12-01T11:08:36.2991667Z Running tests...
2022-12-01T11:08:36.2991932Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2992247Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2992631Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T11:08:36.2992653Z 
2022-12-01T11:08:36.2992933Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2993028Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2993066Z 
2022-12-01T11:08:36.2993155Z OK (skipped=1)
2022-12-01T11:08:36.2993174Z 
2022-12-01T11:08:36.2993298Z Generating XML reports...
2022-12-01T11:08:36.2993747Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105127.xml
2022-12-01T11:08:36.2994117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2994296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2994681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2994882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2994902Z 
2022-12-01T11:08:36.2995010Z Running tests...
2022-12-01T11:08:36.2995257Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2995570Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2995857Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T11:08:36.2995876Z 
2022-12-01T11:08:36.2996139Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2996254Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2996274Z 
2022-12-01T11:08:36.2996383Z OK (skipped=1)
2022-12-01T11:08:36.2996403Z 
2022-12-01T11:08:36.2996529Z Generating XML reports...
2022-12-01T11:08:36.2996975Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105130.xml
2022-12-01T11:08:36.2997429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.2997588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.2997973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.2998165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.2998185Z 
2022-12-01T11:08:36.2998292Z Running tests...
2022-12-01T11:08:36.2998554Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2998867Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.2999166Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T11:08:36.2999190Z 
2022-12-01T11:08:36.2999466Z ----------------------------------------------------------------------
2022-12-01T11:08:36.2999560Z Ran 1 test in 0.002s
2022-12-01T11:08:36.2999599Z 
2022-12-01T11:08:36.2999687Z OK (skipped=1)
2022-12-01T11:08:36.2999706Z 
2022-12-01T11:08:36.2999832Z Generating XML reports...
2022-12-01T11:08:36.3000273Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105132.xml
2022-12-01T11:08:36.3000647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3000822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3001204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3001396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3001416Z 
2022-12-01T11:08:36.3001527Z Running tests...
2022-12-01T11:08:36.3001820Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3002145Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3002569Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3002588Z 
2022-12-01T11:08:36.3002849Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3002960Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3002980Z 
2022-12-01T11:08:36.3003086Z OK (skipped=1)
2022-12-01T11:08:36.3003104Z 
2022-12-01T11:08:36.3003225Z Generating XML reports...
2022-12-01T11:08:36.3003671Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105134.xml
2022-12-01T11:08:36.3004049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3004214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3004597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3004790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3004809Z 
2022-12-01T11:08:36.3004915Z Running tests...
2022-12-01T11:08:36.3005177Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3005490Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3005916Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3005936Z 
2022-12-01T11:08:36.3006196Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3006308Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3006379Z 
2022-12-01T11:08:36.3006472Z OK (skipped=1)
2022-12-01T11:08:36.3006644Z 
2022-12-01T11:08:36.3006779Z Generating XML reports...
2022-12-01T11:08:36.3007406Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105136.xml
2022-12-01T11:08:36.3008123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3008561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3009144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3009621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3009853Z 
2022-12-01T11:08:36.3009964Z Running tests...
2022-12-01T11:08:36.3010358Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3010896Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3011411Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2022-12-01T11:08:36.3011706Z 
2022-12-01T11:08:36.3011955Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3012294Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3012454Z 
2022-12-01T11:08:36.3012568Z OK (skipped=1)
2022-12-01T11:08:36.3012726Z 
2022-12-01T11:08:36.3012852Z Generating XML reports...
2022-12-01T11:08:36.3013442Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105138.xml
2022-12-01T11:08:36.3014163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3014621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3015185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3015718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3015957Z 
2022-12-01T11:08:36.3016072Z Running tests...
2022-12-01T11:08:36.3016484Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3016999Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3017672Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3018016Z 
2022-12-01T11:08:36.3018284Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3018624Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3018767Z 
2022-12-01T11:08:36.3018874Z OK (skipped=1)
2022-12-01T11:08:36.3019033Z 
2022-12-01T11:08:36.3019157Z Generating XML reports...
2022-12-01T11:08:36.3019757Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105140.xml
2022-12-01T11:08:36.3020465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3020924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3021520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3022007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3022245Z 
2022-12-01T11:08:36.3022339Z Running tests...
2022-12-01T11:08:36.3022747Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3023277Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3023950Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3024329Z 
2022-12-01T11:08:36.3024599Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3024922Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3025084Z 
2022-12-01T11:08:36.3025191Z OK (skipped=1)
2022-12-01T11:08:36.3025346Z 
2022-12-01T11:08:36.3025452Z Generating XML reports...
2022-12-01T11:08:36.3026056Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105142.xml
2022-12-01T11:08:36.3026765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3027216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3027781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3028253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3028488Z 
2022-12-01T11:08:36.3028599Z Running tests...
2022-12-01T11:08:36.3028991Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3029785Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3030490Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3030820Z 
2022-12-01T11:08:36.3031085Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3031393Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3031575Z 
2022-12-01T11:08:36.3031690Z OK (skipped=1)
2022-12-01T11:08:36.3031849Z 
2022-12-01T11:08:36.3031974Z Generating XML reports...
2022-12-01T11:08:36.3032563Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105144.xml
2022-12-01T11:08:36.3033287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3033845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3034476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3034947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3035160Z 
2022-12-01T11:08:36.3035271Z Running tests...
2022-12-01T11:08:36.3035677Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3036208Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3036857Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3037181Z 
2022-12-01T11:08:36.3037443Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3037776Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3037939Z 
2022-12-01T11:08:36.3038051Z OK (skipped=1)
2022-12-01T11:08:36.3038188Z 
2022-12-01T11:08:36.3038314Z Generating XML reports...
2022-12-01T11:08:36.3038918Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105146.xml
2022-12-01T11:08:36.3039632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3040068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3040644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3041121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3041352Z 
2022-12-01T11:08:36.3041460Z Running tests...
2022-12-01T11:08:36.3041844Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3042498Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3043181Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3043508Z 
2022-12-01T11:08:36.3043776Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3044084Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3044245Z 
2022-12-01T11:08:36.3044358Z OK (skipped=1)
2022-12-01T11:08:36.3044512Z 
2022-12-01T11:08:36.3044637Z Generating XML reports...
2022-12-01T11:08:36.3045240Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105148.xml
2022-12-01T11:08:36.3045934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3046384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3046973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3047430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3047662Z 
2022-12-01T11:08:36.3047773Z Running tests...
2022-12-01T11:08:36.3048180Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3048711Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3049363Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3049680Z 
2022-12-01T11:08:36.3049949Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3050309Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3050474Z 
2022-12-01T11:08:36.3050602Z OK (skipped=1)
2022-12-01T11:08:36.3050760Z 
2022-12-01T11:08:36.3050873Z Generating XML reports...
2022-12-01T11:08:36.3051550Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105150.xml
2022-12-01T11:08:36.3052312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3052765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3053328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3053824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3054058Z 
2022-12-01T11:08:36.3054168Z Running tests...
2022-12-01T11:08:36.3054582Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3055100Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3055778Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3056110Z 
2022-12-01T11:08:36.3056374Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3056707Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3056849Z 
2022-12-01T11:08:36.3056958Z OK (skipped=1)
2022-12-01T11:08:36.3057109Z 
2022-12-01T11:08:36.3057236Z Generating XML reports...
2022-12-01T11:08:36.3057839Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105152.xml
2022-12-01T11:08:36.3058538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3058989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3059565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3060097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3060329Z 
2022-12-01T11:08:36.3060423Z Running tests...
2022-12-01T11:08:36.3060829Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3061362Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3062001Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3062319Z 
2022-12-01T11:08:36.3062580Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3062904Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3063066Z 
2022-12-01T11:08:36.3063173Z OK (skipped=1)
2022-12-01T11:08:36.3063326Z 
2022-12-01T11:08:36.3063431Z Generating XML reports...
2022-12-01T11:08:36.3064032Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105154.xml
2022-12-01T11:08:36.3064751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3065201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3065756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3066223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3066455Z 
2022-12-01T11:08:36.3066565Z Running tests...
2022-12-01T11:08:36.3066952Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3067483Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3068131Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.003s)
2022-12-01T11:08:36.3068440Z 
2022-12-01T11:08:36.3068705Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3069020Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3069235Z 
2022-12-01T11:08:36.3069576Z OK (skipped=1)
2022-12-01T11:08:36.3069756Z 
2022-12-01T11:08:36.3069882Z Generating XML reports...
2022-12-01T11:08:36.3070517Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105156.xml
2022-12-01T11:08:36.3071239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3071689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3072262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3072714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3072944Z 
2022-12-01T11:08:36.3073051Z Running tests...
2022-12-01T11:08:36.3073457Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3073982Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3074533Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3075060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71197
2022-12-01T11:08:36.3075514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71198
2022-12-01T11:08:36.3076108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3076557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3077134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3077603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3078279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3078724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3079299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3079746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3080206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3080705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3081366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3082041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3082568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3083561Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:08:36.3084195Z   warnings.warn(
2022-12-01T11:08:36.3084548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3085516Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:08:36.3086130Z   warnings.warn(
2022-12-01T11:08:36.3086378Z ok (3.984s)
2022-12-01T11:08:36.3086506Z 
2022-12-01T11:08:36.3086854Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3087266Z Ran 1 test in 3.984s
2022-12-01T11:08:36.3087426Z 
2022-12-01T11:08:36.3087518Z OK
2022-12-01T11:08:36.3087653Z 
2022-12-01T11:08:36.3087759Z Generating XML reports...
2022-12-01T11:08:36.3088367Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105159.xml
2022-12-01T11:08:36.3089081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3089532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3090090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3090564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3090798Z 
2022-12-01T11:08:36.3090907Z Running tests...
2022-12-01T11:08:36.3091297Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3091833Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3092485Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3092793Z 
2022-12-01T11:08:36.3093057Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3093365Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3093524Z 
2022-12-01T11:08:36.3093631Z OK (skipped=1)
2022-12-01T11:08:36.3093783Z 
2022-12-01T11:08:36.3093908Z Generating XML reports...
2022-12-01T11:08:36.3094490Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105205.xml
2022-12-01T11:08:36.3095209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3095729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3096309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3096760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3096989Z 
2022-12-01T11:08:36.3097097Z Running tests...
2022-12-01T11:08:36.3097502Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3098032Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3098675Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3098985Z 
2022-12-01T11:08:36.3099252Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3099579Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3099742Z 
2022-12-01T11:08:36.3099831Z OK (skipped=1)
2022-12-01T11:08:36.3099985Z 
2022-12-01T11:08:36.3100112Z Generating XML reports...
2022-12-01T11:08:36.3100719Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105207.xml
2022-12-01T11:08:36.3101435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3101869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3102440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3102907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3103136Z 
2022-12-01T11:08:36.3103245Z Running tests...
2022-12-01T11:08:36.3103632Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3104165Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3104868Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.3105185Z 
2022-12-01T11:08:36.3105436Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3105762Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3105923Z 
2022-12-01T11:08:36.3106030Z OK (skipped=1)
2022-12-01T11:08:36.3106190Z 
2022-12-01T11:08:36.3106314Z Generating XML reports...
2022-12-01T11:08:36.3106899Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105209.xml
2022-12-01T11:08:36.3107611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3108064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3108620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3109101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3109331Z 
2022-12-01T11:08:36.3109691Z Running tests...
2022-12-01T11:08:36.3110112Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3110618Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3111151Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3111664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71782
2022-12-01T11:08:36.3112094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71783
2022-12-01T11:08:36.3112703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3113251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3113828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3114278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3114854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3115298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3115863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3116307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3116759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3117258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3117907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3118594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3119115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3119576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3119896Z ok (4.201s)
2022-12-01T11:08:36.3120041Z 
2022-12-01T11:08:36.3120306Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3120629Z Ran 1 test in 4.202s
2022-12-01T11:08:36.3120787Z 
2022-12-01T11:08:36.3120863Z OK
2022-12-01T11:08:36.3120994Z 
2022-12-01T11:08:36.3121114Z Generating XML reports...
2022-12-01T11:08:36.3121718Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105211.xml
2022-12-01T11:08:36.3122507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3122955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3123533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3123999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3124227Z 
2022-12-01T11:08:36.3124318Z Running tests...
2022-12-01T11:08:36.3124722Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3125248Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3125759Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3126248Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72129
2022-12-01T11:08:36.3126700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72130
2022-12-01T11:08:36.3127309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3127734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3128311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3128774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3129350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3129774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3130344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3130872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3131325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3131806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3132461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3133150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3133655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3134145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3134626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3135110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3135742Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3136426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3137055Z STAGE:2022-12-01 10:52:21 72129:72129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3137628Z STAGE:2022-12-01 10:52:21 72130:72130 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3138088Z [1669891943.508225] [7ff5451e42d4:72129:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3138568Z [1669891943.508225] [7ff5451e42d4:72129:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3139135Z [1669891943.489089] [7ff5451e42d4:72130:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3139610Z [1669891943.489089] [7ff5451e42d4:72130:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3140384Z STAGE:2022-12-01 10:52:24 72129:72129 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:52:24 72130:72130 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3140771Z 
2022-12-01T11:08:36.3141103Z STAGE:2022-12-01 10:52:24 72129:72129 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3141671Z STAGE:2022-12-01 10:52:24 72130:72130 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3142246Z STAGE:2022-12-01 10:52:24 72129:72129 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3142814Z STAGE:2022-12-01 10:52:24 72130:72130 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3143161Z ok (7.246s)
2022-12-01T11:08:36.3143306Z 
2022-12-01T11:08:36.3143577Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3143886Z Ran 1 test in 7.246s
2022-12-01T11:08:36.3144051Z 
2022-12-01T11:08:36.3144141Z OK
2022-12-01T11:08:36.3144271Z 
2022-12-01T11:08:36.3144392Z Generating XML reports...
2022-12-01T11:08:36.3144991Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105218.xml
2022-12-01T11:08:36.3145690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3146132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3146697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3147250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3147461Z 
2022-12-01T11:08:36.3147573Z Running tests...
2022-12-01T11:08:36.3147977Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3148501Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3148994Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3149761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72541
2022-12-01T11:08:36.3150222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72542
2022-12-01T11:08:36.3150838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3151268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3151844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3152305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3152862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3153305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3153877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3154338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3154770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3155265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3155920Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3156697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3157214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3157693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3158171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3158632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3159291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3159972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3160605Z STAGE:2022-12-01 10:52:31 72541:72541 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3161156Z STAGE:2022-12-01 10:52:31 72542:72542 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3161627Z [1669891953.022360] [7ff5451e42d4:72541:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3162099Z [1669891953.022360] [7ff5451e42d4:72541:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3162563Z [1669891953.003521] [7ff5451e42d4:72542:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3163016Z [1669891953.003521] [7ff5451e42d4:72542:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3163816Z STAGE:2022-12-01 10:52:33 72541:72541 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:52:33 72542:72542 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3164286Z 
2022-12-01T11:08:36.3164622Z STAGE:2022-12-01 10:52:33 72542:72542 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3165190Z STAGE:2022-12-01 10:52:33 72541:72541 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3165750Z STAGE:2022-12-01 10:52:33 72542:72542 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3166334Z STAGE:2022-12-01 10:52:33 72541:72541 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3166679Z ok (7.239s)
2022-12-01T11:08:36.3166826Z 
2022-12-01T11:08:36.3167097Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3167406Z Ran 1 test in 7.239s
2022-12-01T11:08:36.3167566Z 
2022-12-01T11:08:36.3167657Z OK
2022-12-01T11:08:36.3167788Z 
2022-12-01T11:08:36.3167912Z Generating XML reports...
2022-12-01T11:08:36.3168504Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105227.xml
2022-12-01T11:08:36.3169222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3169673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3170251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3170754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3170980Z 
2022-12-01T11:08:36.3171088Z Running tests...
2022-12-01T11:08:36.3171493Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3172004Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3172526Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3173089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72926
2022-12-01T11:08:36.3173553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72927
2022-12-01T11:08:36.3174147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3174588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3175169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3175618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3176194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3176637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3177214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3177660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3178105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3178602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3179262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3179932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3180452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3180933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3181462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3181936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3182595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3183226Z STAGE:2022-12-01 10:52:40 72926:72926 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3183853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3184484Z STAGE:2022-12-01 10:52:40 72927:72927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3184963Z [1669891962.300072] [7ff5451e42d4:72926:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3185450Z [1669891962.300072] [7ff5451e42d4:72926:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3185898Z [1669891962.281661] [7ff5451e42d4:72927:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3186369Z [1669891962.281661] [7ff5451e42d4:72927:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3187161Z STAGE:2022-12-01 10:52:42 72926:72926 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:52:42 72927:72927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3187549Z 
2022-12-01T11:08:36.3187882Z STAGE:2022-12-01 10:52:42 72926:72926 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3188434Z STAGE:2022-12-01 10:52:42 72927:72927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3189067Z STAGE:2022-12-01 10:52:42 72927:72927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3189875Z STAGE:2022-12-01 10:52:42 72926:72926 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3190228Z ok (7.138s)
2022-12-01T11:08:36.3190357Z 
2022-12-01T11:08:36.3190632Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3190957Z Ran 1 test in 7.138s
2022-12-01T11:08:36.3191118Z 
2022-12-01T11:08:36.3191209Z OK
2022-12-01T11:08:36.3191342Z 
2022-12-01T11:08:36.3191449Z Generating XML reports...
2022-12-01T11:08:36.3192055Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105236.xml
2022-12-01T11:08:36.3192769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3193222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3193793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3194261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3194493Z 
2022-12-01T11:08:36.3194599Z Running tests...
2022-12-01T11:08:36.3194989Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3195520Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3196035Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3196530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73357
2022-12-01T11:08:36.3196961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73358
2022-12-01T11:08:36.3197561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3198101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3198686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3199142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3199728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3200169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3200723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3201186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3201638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3202132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3202779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3203468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3203983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3204462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3204920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3205397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3206056Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3206749Z STAGE:2022-12-01 10:52:50 73357:73357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3207410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3208034Z STAGE:2022-12-01 10:52:50 73358:73358 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3208514Z [1669891971.780022] [7ff5451e42d4:73358:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3208974Z [1669891971.780022] [7ff5451e42d4:73358:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3209446Z [1669891971.704259] [7ff5451e42d4:73357:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3209907Z [1669891971.704259] [7ff5451e42d4:73357:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3210709Z STAGE:2022-12-01 10:52:52 73358:73358 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:52:52 73357:73357 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3211106Z 
2022-12-01T11:08:36.3211442Z STAGE:2022-12-01 10:52:52 73358:73358 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3211996Z STAGE:2022-12-01 10:52:52 73357:73357 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3212572Z STAGE:2022-12-01 10:52:52 73358:73358 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3213151Z STAGE:2022-12-01 10:52:52 73357:73357 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3213476Z ok (7.159s)
2022-12-01T11:08:36.3213626Z 
2022-12-01T11:08:36.3213897Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3214224Z Ran 1 test in 7.159s
2022-12-01T11:08:36.3214454Z 
2022-12-01T11:08:36.3214546Z OK
2022-12-01T11:08:36.3214661Z 
2022-12-01T11:08:36.3214786Z Generating XML reports...
2022-12-01T11:08:36.3215393Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105246.xml
2022-12-01T11:08:36.3216111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3216544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3217125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3217599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3217826Z 
2022-12-01T11:08:36.3217934Z Running tests...
2022-12-01T11:08:36.3218320Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3218847Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3219365Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3219840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73821
2022-12-01T11:08:36.3220290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73822
2022-12-01T11:08:36.3220908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3221360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3221918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3222392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3222967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3223487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3224128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3224597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3225048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3225530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3226198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3226890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3227400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3227865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3228265Z skip: Skipped due to small world size. (3.910s)
2022-12-01T11:08:36.3228465Z 
2022-12-01T11:08:36.3228743Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3229074Z Ran 1 test in 3.910s
2022-12-01T11:08:36.3229216Z 
2022-12-01T11:08:36.3229323Z OK (skipped=1)
2022-12-01T11:08:36.3229720Z 
2022-12-01T11:08:36.3229849Z Generating XML reports...
2022-12-01T11:08:36.3230465Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105255.xml
2022-12-01T11:08:36.3231166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3231621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3232202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3232777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3232990Z 
2022-12-01T11:08:36.3233101Z Running tests...
2022-12-01T11:08:36.3233511Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3234051Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3234540Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3235035Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74105
2022-12-01T11:08:36.3235486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74106
2022-12-01T11:08:36.3236098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3236528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3237116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3237588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3238151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3238603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3239184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3239650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3240085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3240588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3241323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3242039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3242538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3243011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3243409Z skip: Skipped due to small world size. (3.863s)
2022-12-01T11:08:36.3243612Z 
2022-12-01T11:08:36.3243883Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3244197Z Ran 1 test in 3.863s
2022-12-01T11:08:36.3244363Z 
2022-12-01T11:08:36.3244467Z OK (skipped=1)
2022-12-01T11:08:36.3244621Z 
2022-12-01T11:08:36.3244747Z Generating XML reports...
2022-12-01T11:08:36.3245339Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105301.xml
2022-12-01T11:08:36.3246063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3246514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3247096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3247553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3247782Z 
2022-12-01T11:08:36.3247890Z Running tests...
2022-12-01T11:08:36.3248298Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3248811Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3249332Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3249904Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74416
2022-12-01T11:08:36.3250360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74417
2022-12-01T11:08:36.3250957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3251408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3251987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3252439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3253024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3253472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3254045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3254498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3254955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3255448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3256107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3256784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3257310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3257789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3258168Z skip: Skipped due to small world size. (3.974s)
2022-12-01T11:08:36.3258374Z 
2022-12-01T11:08:36.3258698Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3258819Z Ran 1 test in 3.974s
2022-12-01T11:08:36.3258839Z 
2022-12-01T11:08:36.3258947Z OK (skipped=1)
2022-12-01T11:08:36.3258966Z 
2022-12-01T11:08:36.3259090Z Generating XML reports...
2022-12-01T11:08:36.3259550Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105307.xml
2022-12-01T11:08:36.3259934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3260110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3260477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3260668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3260692Z 
2022-12-01T11:08:36.3260805Z Running tests...
2022-12-01T11:08:36.3261073Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3261386Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3261650Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3261876Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74731
2022-12-01T11:08:36.3262093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74732
2022-12-01T11:08:36.3262452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3262635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3263017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3263296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3263667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3263846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3264223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3264412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3264658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3264883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3265292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3265702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3265938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3266171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3266329Z skip: Skipped due to small world size. (3.875s)
2022-12-01T11:08:36.3266348Z 
2022-12-01T11:08:36.3266615Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3266727Z Ran 1 test in 3.875s
2022-12-01T11:08:36.3266746Z 
2022-12-01T11:08:36.3266851Z OK (skipped=1)
2022-12-01T11:08:36.3266871Z 
2022-12-01T11:08:36.3266976Z Generating XML reports...
2022-12-01T11:08:36.3267423Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105314.xml
2022-12-01T11:08:36.3267801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3268029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3268431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3268622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3268641Z 
2022-12-01T11:08:36.3268756Z Running tests...
2022-12-01T11:08:36.3269024Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3269340Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3269820Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3270044Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75046
2022-12-01T11:08:36.3270262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75047
2022-12-01T11:08:36.3270691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3270869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3271256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3271450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3271815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3271971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3272350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3272541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3272890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3273135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3273544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3273941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3274177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3274519Z STAGE:2022-12-01 10:53:23 75047:75047 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3274729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3275069Z STAGE:2022-12-01 10:53:23 75046:75046 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3275312Z [1669892005.640023] [7ff5451e42d4:75046:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3275553Z [1669892005.640023] [7ff5451e42d4:75046:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3275789Z [1669892005.643190] [7ff5451e42d4:75047:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3276018Z [1669892005.643190] [7ff5451e42d4:75047:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3276575Z STAGE:2022-12-01 10:53:26 75046:75046 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:53:26 75047:75047 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3276596Z 
2022-12-01T11:08:36.3276925Z STAGE:2022-12-01 10:53:26 75047:75047 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3277325Z STAGE:2022-12-01 10:53:26 75046:75046 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3277684Z STAGE:2022-12-01 10:53:26 75047:75047 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3278000Z STAGE:2022-12-01 10:53:26 75046:75046 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3278102Z ok (6.999s)
2022-12-01T11:08:36.3278122Z 
2022-12-01T11:08:36.3278388Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3278504Z Ran 1 test in 6.999s
2022-12-01T11:08:36.3278523Z 
2022-12-01T11:08:36.3278617Z OK
2022-12-01T11:08:36.3278636Z 
2022-12-01T11:08:36.3278759Z Generating XML reports...
2022-12-01T11:08:36.3279208Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105320.xml
2022-12-01T11:08:36.3279593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3279776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3280138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3280338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3280357Z 
2022-12-01T11:08:36.3280465Z Running tests...
2022-12-01T11:08:36.3280729Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3281048Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3281303Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3281529Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75472
2022-12-01T11:08:36.3281746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75473
2022-12-01T11:08:36.3282170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3282352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3282735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3282925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3283295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3283465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3283834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3284027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3284277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3284505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3284909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3285312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3285548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3285775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3286111Z STAGE:2022-12-01 10:53:33 75473:75473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3286440Z STAGE:2022-12-01 10:53:33 75472:75472 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3286733Z [1669892014.921614] [7ff5451e42d4:75472:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3286988Z [1669892014.921614] [7ff5451e42d4:75472:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3287200Z [1669892014.958645] [7ff5451e42d4:75473:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3287438Z [1669892014.958645] [7ff5451e42d4:75473:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3288001Z STAGE:2022-12-01 10:53:35 75472:75472 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:53:35 75473:75473 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3288022Z 
2022-12-01T11:08:36.3288357Z STAGE:2022-12-01 10:53:35 75473:75473 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3288695Z STAGE:2022-12-01 10:53:35 75472:75472 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3289031Z STAGE:2022-12-01 10:53:35 75473:75473 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3289362Z STAGE:2022-12-01 10:53:35 75472:75472 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3289472Z ok (6.937s)
2022-12-01T11:08:36.3289492Z 
2022-12-01T11:08:36.3289765Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3289877Z Ran 1 test in 6.938s
2022-12-01T11:08:36.3289896Z 
2022-12-01T11:08:36.3289970Z OK
2022-12-01T11:08:36.3290006Z 
2022-12-01T11:08:36.3290111Z Generating XML reports...
2022-12-01T11:08:36.3290559Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105329.xml
2022-12-01T11:08:36.3290936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3291180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3291567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3291759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3291778Z 
2022-12-01T11:08:36.3291890Z Running tests...
2022-12-01T11:08:36.3292155Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3292451Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3292724Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: UCC all_reduce multigpu skipped (0.002s)
2022-12-01T11:08:36.3292745Z 
2022-12-01T11:08:36.3293010Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3293122Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3293145Z 
2022-12-01T11:08:36.3293252Z OK (skipped=1)
2022-12-01T11:08:36.3293271Z 
2022-12-01T11:08:36.3293396Z Generating XML reports...
2022-12-01T11:08:36.3293849Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105338.xml
2022-12-01T11:08:36.3294230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3294405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3294771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3294968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3294987Z 
2022-12-01T11:08:36.3295095Z Running tests...
2022-12-01T11:08:36.3295363Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3295676Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3296010Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: UCC all_reduce multigpu skipped (0.002s)
2022-12-01T11:08:36.3296032Z 
2022-12-01T11:08:36.3296307Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3296421Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3296441Z 
2022-12-01T11:08:36.3296547Z OK (skipped=1)
2022-12-01T11:08:36.3296566Z 
2022-12-01T11:08:36.3296672Z Generating XML reports...
2022-12-01T11:08:36.3297121Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105340.xml
2022-12-01T11:08:36.3297500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3297676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3298055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3298255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3298274Z 
2022-12-01T11:08:36.3298381Z Running tests...
2022-12-01T11:08:36.3298648Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3298942Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3299206Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3299422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76063
2022-12-01T11:08:36.3299645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76064
2022-12-01T11:08:36.3300023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3300199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3300651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3300849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3301217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3301374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3301755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3301943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3302193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3302437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3302846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3303254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3303486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3303829Z STAGE:2022-12-01 10:53:46 76064:76064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3304038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3304362Z STAGE:2022-12-01 10:53:46 76063:76063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3304603Z [1669892028.347830] [7ff5451e42d4:76063:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3304843Z [1669892028.347830] [7ff5451e42d4:76063:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3305127Z [1669892028.348492] [7ff5451e42d4:76064:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3305377Z [1669892028.348492] [7ff5451e42d4:76064:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3305939Z STAGE:2022-12-01 10:53:48 76063:76063 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:53:48 76064:76064 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3305960Z 
2022-12-01T11:08:36.3306290Z STAGE:2022-12-01 10:53:48 76064:76064 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3306612Z STAGE:2022-12-01 10:53:48 76063:76063 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3306954Z STAGE:2022-12-01 10:53:48 76064:76064 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3307277Z STAGE:2022-12-01 10:53:48 76063:76063 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3307388Z ok (7.027s)
2022-12-01T11:08:36.3307407Z 
2022-12-01T11:08:36.3307673Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3307785Z Ran 1 test in 7.027s
2022-12-01T11:08:36.3307804Z 
2022-12-01T11:08:36.3307894Z OK
2022-12-01T11:08:36.3307913Z 
2022-12-01T11:08:36.3308041Z Generating XML reports...
2022-12-01T11:08:36.3308492Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105343.xml
2022-12-01T11:08:36.3308866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3309051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3309654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3309950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3309970Z 
2022-12-01T11:08:36.3310080Z Running tests...
2022-12-01T11:08:36.3310362Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3310676Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3311085Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T11:08:36.3311106Z 
2022-12-01T11:08:36.3311386Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3311503Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3311522Z 
2022-12-01T11:08:36.3311630Z OK (skipped=1)
2022-12-01T11:08:36.3311648Z 
2022-12-01T11:08:36.3311753Z Generating XML reports...
2022-12-01T11:08:36.3312202Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105352.xml
2022-12-01T11:08:36.3312593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3312769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3313155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3313347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3313367Z 
2022-12-01T11:08:36.3313475Z Running tests...
2022-12-01T11:08:36.3313743Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3314035Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3314286Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3314513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76602
2022-12-01T11:08:36.3314794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76603
2022-12-01T11:08:36.3315196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3315373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3315761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3315950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3316315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3316471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3316851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3317047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3317297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3317542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3317953Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3318357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3318587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3318931Z STAGE:2022-12-01 10:53:57 76602:76602 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3319137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3319529Z STAGE:2022-12-01 10:53:58 76603:76603 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3319766Z [1669892039.929276] [7ff5451e42d4:76602:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3320008Z [1669892039.929276] [7ff5451e42d4:76602:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3320244Z [1669892039.979230] [7ff5451e42d4:76603:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3320480Z [1669892039.979230] [7ff5451e42d4:76603:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3321039Z STAGE:2022-12-01 10:54:00 76602:76602 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:54:00 76603:76603 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3321065Z 
2022-12-01T11:08:36.3321408Z STAGE:2022-12-01 10:54:00 76603:76603 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3321731Z STAGE:2022-12-01 10:54:00 76602:76602 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3322278Z STAGE:2022-12-01 10:54:00 76602:76602 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:54:00 76603:76603 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3322298Z 
2022-12-01T11:08:36.3322400Z ok (7.120s)
2022-12-01T11:08:36.3322420Z 
2022-12-01T11:08:36.3322668Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3322787Z Ran 1 test in 7.120s
2022-12-01T11:08:36.3322806Z 
2022-12-01T11:08:36.3322898Z OK
2022-12-01T11:08:36.3322917Z 
2022-12-01T11:08:36.3323041Z Generating XML reports...
2022-12-01T11:08:36.3323496Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105354.xml
2022-12-01T11:08:36.3323919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3324103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3324491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3324665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3324704Z 
2022-12-01T11:08:36.3324793Z Running tests...
2022-12-01T11:08:36.3325064Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3325376Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3325636Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3325866Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77021
2022-12-01T11:08:36.3326088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77022
2022-12-01T11:08:36.3326459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3326641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3327002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3327191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3327559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3327732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3328111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3328377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3328629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3328880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3329284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3329664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3329901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3330242Z STAGE:2022-12-01 10:54:07 77021:77021 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3330468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3330811Z STAGE:2022-12-01 10:54:07 77022:77022 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3331048Z [1669892049.231911] [7ff5451e42d4:77022:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3331292Z [1669892049.231911] [7ff5451e42d4:77022:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3331527Z [1669892049.231979] [7ff5451e42d4:77021:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3331765Z [1669892049.231979] [7ff5451e42d4:77021:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3332296Z STAGE:2022-12-01 10:54:09 77022:77022 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:54:09 77021:77021 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3332344Z 
2022-12-01T11:08:36.3332704Z STAGE:2022-12-01 10:54:09 77022:77022 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3333045Z STAGE:2022-12-01 10:54:09 77021:77021 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3333384Z STAGE:2022-12-01 10:54:09 77022:77022 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3333722Z STAGE:2022-12-01 10:54:09 77021:77021 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3333827Z ok (7.182s)
2022-12-01T11:08:36.3333848Z 
2022-12-01T11:08:36.3334117Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3334234Z Ran 1 test in 7.182s
2022-12-01T11:08:36.3334253Z 
2022-12-01T11:08:36.3334347Z OK
2022-12-01T11:08:36.3334367Z 
2022-12-01T11:08:36.3334491Z Generating XML reports...
2022-12-01T11:08:36.3334924Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105403.xml
2022-12-01T11:08:36.3335310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3335494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3335877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3336067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3336087Z 
2022-12-01T11:08:36.3336194Z Running tests...
2022-12-01T11:08:36.3336466Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3336775Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3337024Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3337338Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77439
2022-12-01T11:08:36.3337565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77440
2022-12-01T11:08:36.3337945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3338131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3338514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3338703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3339070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3339242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3339599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3339797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3340042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3340292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3340699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3341096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3341327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3341662Z STAGE:2022-12-01 10:54:16 77439:77439 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3341894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3342265Z STAGE:2022-12-01 10:54:16 77440:77440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3342515Z [1669892058.665235] [7ff5451e42d4:77440:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3342761Z [1669892058.665235] [7ff5451e42d4:77440:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3342990Z [1669892058.707809] [7ff5451e42d4:77439:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3343229Z [1669892058.707809] [7ff5451e42d4:77439:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3343784Z STAGE:2022-12-01 10:54:19 77440:77440 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:54:19 77439:77439 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3343809Z 
2022-12-01T11:08:36.3344146Z STAGE:2022-12-01 10:54:19 77440:77440 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3344482Z STAGE:2022-12-01 10:54:19 77439:77439 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3344818Z STAGE:2022-12-01 10:54:19 77440:77440 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3345155Z STAGE:2022-12-01 10:54:19 77439:77439 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3345238Z ok (7.131s)
2022-12-01T11:08:36.3345257Z 
2022-12-01T11:08:36.3345524Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3345638Z Ran 1 test in 7.132s
2022-12-01T11:08:36.3345659Z 
2022-12-01T11:08:36.3345751Z OK
2022-12-01T11:08:36.3345770Z 
2022-12-01T11:08:36.3345895Z Generating XML reports...
2022-12-01T11:08:36.3346346Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105413.xml
2022-12-01T11:08:36.3346796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3346975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3347358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3347531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3347551Z 
2022-12-01T11:08:36.3347664Z Running tests...
2022-12-01T11:08:36.3347931Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3348248Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3348546Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-12-01T11:08:36.3348570Z 
2022-12-01T11:08:36.3348835Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3348946Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3348966Z 
2022-12-01T11:08:36.3349076Z OK (skipped=1)
2022-12-01T11:08:36.3349096Z 
2022-12-01T11:08:36.3349224Z Generating XML reports...
2022-12-01T11:08:36.3349960Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105422.xml
2022-12-01T11:08:36.3350347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3350533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3350917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3351107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3351131Z 
2022-12-01T11:08:36.3351238Z Running tests...
2022-12-01T11:08:36.3351589Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3351917Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3352204Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-12-01T11:08:36.3352249Z 
2022-12-01T11:08:36.3352493Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3352606Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3352625Z 
2022-12-01T11:08:36.3352731Z OK (skipped=1)
2022-12-01T11:08:36.3352750Z 
2022-12-01T11:08:36.3352873Z Generating XML reports...
2022-12-01T11:08:36.3353318Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105424.xml
2022-12-01T11:08:36.3353703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3353885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3354272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3354446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3354482Z 
2022-12-01T11:08:36.3354573Z Running tests...
2022-12-01T11:08:36.3354846Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3355158Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3355473Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and NCCL backends will have CUDA allReduce tested (0.002s)
2022-12-01T11:08:36.3355493Z 
2022-12-01T11:08:36.3355752Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3355939Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3355958Z 
2022-12-01T11:08:36.3356068Z OK (skipped=1)
2022-12-01T11:08:36.3356087Z 
2022-12-01T11:08:36.3356216Z Generating XML reports...
2022-12-01T11:08:36.3356646Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105426.xml
2022-12-01T11:08:36.3357025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3357203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3357578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3357769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3357788Z 
2022-12-01T11:08:36.3357901Z Running tests...
2022-12-01T11:08:36.3358163Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3358485Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3358731Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:08:36.3358751Z 
2022-12-01T11:08:36.3358992Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3359111Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3359130Z 
2022-12-01T11:08:36.3359237Z OK (skipped=1)
2022-12-01T11:08:36.3359255Z 
2022-12-01T11:08:36.3359379Z Generating XML reports...
2022-12-01T11:08:36.3359821Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105428.xml
2022-12-01T11:08:36.3360197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3360372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3360803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3361005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3361024Z 
2022-12-01T11:08:36.3361113Z Running tests...
2022-12-01T11:08:36.3361380Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3361691Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3361947Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:08:36.3361966Z 
2022-12-01T11:08:36.3362227Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3362338Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3362356Z 
2022-12-01T11:08:36.3362462Z OK (skipped=1)
2022-12-01T11:08:36.3362481Z 
2022-12-01T11:08:36.3362604Z Generating XML reports...
2022-12-01T11:08:36.3363058Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105430.xml
2022-12-01T11:08:36.3363422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3363600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3363982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3364174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3364193Z 
2022-12-01T11:08:36.3364303Z Running tests...
2022-12-01T11:08:36.3364569Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3364881Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3365145Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T11:08:36.3365216Z 
2022-12-01T11:08:36.3365492Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3365586Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3365605Z 
2022-12-01T11:08:36.3365712Z OK (skipped=1)
2022-12-01T11:08:36.3365730Z 
2022-12-01T11:08:36.3365854Z Generating XML reports...
2022-12-01T11:08:36.3366303Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105433.xml
2022-12-01T11:08:36.3366680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3366858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3367241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3367431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3367455Z 
2022-12-01T11:08:36.3367563Z Running tests...
2022-12-01T11:08:36.3367814Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3368130Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3368405Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T11:08:36.3368424Z 
2022-12-01T11:08:36.3368689Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3368803Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3368822Z 
2022-12-01T11:08:36.3368929Z OK (skipped=1)
2022-12-01T11:08:36.3368948Z 
2022-12-01T11:08:36.3369071Z Generating XML reports...
2022-12-01T11:08:36.3369518Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105435.xml
2022-12-01T11:08:36.3369872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3370100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3370494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3370735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3370755Z 
2022-12-01T11:08:36.3370864Z Running tests...
2022-12-01T11:08:36.3371127Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3371441Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3371702Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:08:36.3371721Z 
2022-12-01T11:08:36.3371988Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3372082Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3372125Z 
2022-12-01T11:08:36.3372215Z OK (skipped=1)
2022-12-01T11:08:36.3372234Z 
2022-12-01T11:08:36.3372357Z Generating XML reports...
2022-12-01T11:08:36.3372808Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105437.xml
2022-12-01T11:08:36.3373184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3373360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3373745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3373938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3373958Z 
2022-12-01T11:08:36.3374067Z Running tests...
2022-12-01T11:08:36.3374314Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3374629Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3374972Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.003s)
2022-12-01T11:08:36.3374992Z 
2022-12-01T11:08:36.3375257Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3375369Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3375388Z 
2022-12-01T11:08:36.3375498Z OK (skipped=1)
2022-12-01T11:08:36.3375517Z 
2022-12-01T11:08:36.3375641Z Generating XML reports...
2022-12-01T11:08:36.3376090Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105439.xml
2022-12-01T11:08:36.3376465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3376623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3377009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3377208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3377227Z 
2022-12-01T11:08:36.3377335Z Running tests...
2022-12-01T11:08:36.3377601Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3377916Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3378172Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:08:36.3378192Z 
2022-12-01T11:08:36.3378450Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3378564Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3378583Z 
2022-12-01T11:08:36.3378672Z OK (skipped=1)
2022-12-01T11:08:36.3378691Z 
2022-12-01T11:08:36.3378813Z Generating XML reports...
2022-12-01T11:08:36.3379255Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105441.xml
2022-12-01T11:08:36.3379680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3379863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3380246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3380438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3380462Z 
2022-12-01T11:08:36.3380572Z Running tests...
2022-12-01T11:08:36.3380818Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3381133Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3381406Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3381430Z 
2022-12-01T11:08:36.3381691Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3381808Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3381828Z 
2022-12-01T11:08:36.3381933Z OK (skipped=1)
2022-12-01T11:08:36.3381952Z 
2022-12-01T11:08:36.3382075Z Generating XML reports...
2022-12-01T11:08:36.3382522Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105443.xml
2022-12-01T11:08:36.3382896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3383055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3383432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3383624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3383643Z 
2022-12-01T11:08:36.3383810Z Running tests...
2022-12-01T11:08:36.3384077Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3384389Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3384677Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3384697Z 
2022-12-01T11:08:36.3384961Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3385075Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3385094Z 
2022-12-01T11:08:36.3385183Z OK (skipped=1)
2022-12-01T11:08:36.3385201Z 
2022-12-01T11:08:36.3385325Z Generating XML reports...
2022-12-01T11:08:36.3385767Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105445.xml
2022-12-01T11:08:36.3386138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3386317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3386701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3386893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3386913Z 
2022-12-01T11:08:36.3387024Z Running tests...
2022-12-01T11:08:36.3387291Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3387588Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3387884Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3387904Z 
2022-12-01T11:08:36.3388163Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3388275Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3388298Z 
2022-12-01T11:08:36.3388409Z OK (skipped=1)
2022-12-01T11:08:36.3388427Z 
2022-12-01T11:08:36.3388598Z Generating XML reports...
2022-12-01T11:08:36.3389055Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105447.xml
2022-12-01T11:08:36.3389668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3389854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3390229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3390421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3390442Z 
2022-12-01T11:08:36.3390549Z Running tests...
2022-12-01T11:08:36.3390815Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3391126Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3391430Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3391450Z 
2022-12-01T11:08:36.3391714Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3391829Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3391848Z 
2022-12-01T11:08:36.3391937Z OK (skipped=1)
2022-12-01T11:08:36.3391975Z 
2022-12-01T11:08:36.3392079Z Generating XML reports...
2022-12-01T11:08:36.3392527Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105449.xml
2022-12-01T11:08:36.3392897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3393073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3393459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3393756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3393776Z 
2022-12-01T11:08:36.3393885Z Running tests...
2022-12-01T11:08:36.3394151Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3394448Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3394752Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3394773Z 
2022-12-01T11:08:36.3395037Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3395149Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3395168Z 
2022-12-01T11:08:36.3395276Z OK (skipped=1)
2022-12-01T11:08:36.3395296Z 
2022-12-01T11:08:36.3395419Z Generating XML reports...
2022-12-01T11:08:36.3395868Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105452.xml
2022-12-01T11:08:36.3396243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3396420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3396780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3396975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3396995Z 
2022-12-01T11:08:36.3397104Z Running tests...
2022-12-01T11:08:36.3397364Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3397676Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3397969Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3397993Z 
2022-12-01T11:08:36.3398320Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3398442Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3398461Z 
2022-12-01T11:08:36.3398569Z OK (skipped=1)
2022-12-01T11:08:36.3398587Z 
2022-12-01T11:08:36.3398693Z Generating XML reports...
2022-12-01T11:08:36.3399144Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105454.xml
2022-12-01T11:08:36.3399514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3399689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3400069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3400261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3400284Z 
2022-12-01T11:08:36.3400391Z Running tests...
2022-12-01T11:08:36.3400664Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3400961Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3401271Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3401295Z 
2022-12-01T11:08:36.3401556Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3401672Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3401693Z 
2022-12-01T11:08:36.3401796Z OK (skipped=1)
2022-12-01T11:08:36.3401817Z 
2022-12-01T11:08:36.3401940Z Generating XML reports...
2022-12-01T11:08:36.3402383Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105456.xml
2022-12-01T11:08:36.3402754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3402992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3403378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3403552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3403573Z 
2022-12-01T11:08:36.3403680Z Running tests...
2022-12-01T11:08:36.3403941Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3404252Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3404542Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3404562Z 
2022-12-01T11:08:36.3404827Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3404943Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3404962Z 
2022-12-01T11:08:36.3405072Z OK (skipped=1)
2022-12-01T11:08:36.3405091Z 
2022-12-01T11:08:36.3405197Z Generating XML reports...
2022-12-01T11:08:36.3405643Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105458.xml
2022-12-01T11:08:36.3406015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3406190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3406569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3406760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3406779Z 
2022-12-01T11:08:36.3406889Z Running tests...
2022-12-01T11:08:36.3407156Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3407521Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3407813Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3407852Z 
2022-12-01T11:08:36.3408100Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3408212Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3408232Z 
2022-12-01T11:08:36.3408338Z OK (skipped=1)
2022-12-01T11:08:36.3408356Z 
2022-12-01T11:08:36.3408482Z Generating XML reports...
2022-12-01T11:08:36.3408919Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105500.xml
2022-12-01T11:08:36.3409290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3409471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3409855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3410030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3410049Z 
2022-12-01T11:08:36.3410157Z Running tests...
2022-12-01T11:08:36.3410421Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3410730Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3411017Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3411037Z 
2022-12-01T11:08:36.3411300Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3411415Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3411434Z 
2022-12-01T11:08:36.3411542Z OK (skipped=1)
2022-12-01T11:08:36.3411561Z 
2022-12-01T11:08:36.3411740Z Generating XML reports...
2022-12-01T11:08:36.3412175Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105502.xml
2022-12-01T11:08:36.3412546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3412720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3413096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3413286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3413305Z 
2022-12-01T11:08:36.3413416Z Running tests...
2022-12-01T11:08:36.3413677Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3413989Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3414273Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3414316Z 
2022-12-01T11:08:36.3414560Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3414674Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3414693Z 
2022-12-01T11:08:36.3414802Z OK (skipped=1)
2022-12-01T11:08:36.3414821Z 
2022-12-01T11:08:36.3414944Z Generating XML reports...
2022-12-01T11:08:36.3415390Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105504.xml
2022-12-01T11:08:36.3415762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3415937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3416318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3416497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3416582Z 
2022-12-01T11:08:36.3416680Z Running tests...
2022-12-01T11:08:36.3416949Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3417260Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3417553Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3417575Z 
2022-12-01T11:08:36.3417833Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3417945Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3417964Z 
2022-12-01T11:08:36.3418073Z OK (skipped=1)
2022-12-01T11:08:36.3418092Z 
2022-12-01T11:08:36.3418218Z Generating XML reports...
2022-12-01T11:08:36.3418640Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105506.xml
2022-12-01T11:08:36.3419021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3419194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3419570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3419762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3419781Z 
2022-12-01T11:08:36.3419889Z Running tests...
2022-12-01T11:08:36.3420155Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3420465Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3420772Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3420792Z 
2022-12-01T11:08:36.3421097Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3421213Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3421233Z 
2022-12-01T11:08:36.3421339Z OK (skipped=1)
2022-12-01T11:08:36.3421358Z 
2022-12-01T11:08:36.3421481Z Generating XML reports...
2022-12-01T11:08:36.3421921Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105508.xml
2022-12-01T11:08:36.3422291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3422467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3422844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3423035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3423055Z 
2022-12-01T11:08:36.3423144Z Running tests...
2022-12-01T11:08:36.3423416Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3423738Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3424039Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3424059Z 
2022-12-01T11:08:36.3424321Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3424434Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3424453Z 
2022-12-01T11:08:36.3424560Z OK (skipped=1)
2022-12-01T11:08:36.3424579Z 
2022-12-01T11:08:36.3424702Z Generating XML reports...
2022-12-01T11:08:36.3425128Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105510.xml
2022-12-01T11:08:36.3425504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3425685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3426115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3426313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3426333Z 
2022-12-01T11:08:36.3426442Z Running tests...
2022-12-01T11:08:36.3426710Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3427022Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3427332Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:08:36.3427352Z 
2022-12-01T11:08:36.3427618Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3427713Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3427737Z 
2022-12-01T11:08:36.3427847Z OK (skipped=1)
2022-12-01T11:08:36.3427867Z 
2022-12-01T11:08:36.3427991Z Generating XML reports...
2022-12-01T11:08:36.3428435Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105512.xml
2022-12-01T11:08:36.3428808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3428982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3429592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3429806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3429827Z 
2022-12-01T11:08:36.3429917Z Running tests...
2022-12-01T11:08:36.3430189Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3430498Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3430885Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:08:36.3430905Z 
2022-12-01T11:08:36.3431168Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3431280Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3431300Z 
2022-12-01T11:08:36.3431408Z OK (skipped=1)
2022-12-01T11:08:36.3431427Z 
2022-12-01T11:08:36.3431548Z Generating XML reports...
2022-12-01T11:08:36.3432001Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105514.xml
2022-12-01T11:08:36.3432361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3432537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3432919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3433119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3433139Z 
2022-12-01T11:08:36.3433249Z Running tests...
2022-12-01T11:08:36.3433515Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3433828Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3434134Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.003s)
2022-12-01T11:08:36.3434154Z 
2022-12-01T11:08:36.3434420Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3434515Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3434533Z 
2022-12-01T11:08:36.3434641Z OK (skipped=1)
2022-12-01T11:08:36.3434660Z 
2022-12-01T11:08:36.3434781Z Generating XML reports...
2022-12-01T11:08:36.3435283Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105517.xml
2022-12-01T11:08:36.3435674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3435846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3436222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3436414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3436434Z 
2022-12-01T11:08:36.3436541Z Running tests...
2022-12-01T11:08:36.3436787Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3437098Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3437363Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3437589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80418
2022-12-01T11:08:36.3437808Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80419
2022-12-01T11:08:36.3438181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3438358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3438742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3438915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3439280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3439452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3439827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3440081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3440330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3440737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3440980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3441379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3441594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3441823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3442059Z [1669892125.420622] [7ff5451e42d4:80419:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3442301Z [1669892125.420622] [7ff5451e42d4:80419:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3442524Z [1669892125.413362] [7ff5451e42d4:80418:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3442755Z [1669892125.413362] [7ff5451e42d4:80418:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3443002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3443241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3443644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3444080Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3444190Z ok (7.253s)
2022-12-01T11:08:36.3444210Z 
2022-12-01T11:08:36.3444480Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3444591Z Ran 1 test in 7.253s
2022-12-01T11:08:36.3444610Z 
2022-12-01T11:08:36.3444702Z OK
2022-12-01T11:08:36.3444721Z 
2022-12-01T11:08:36.3444844Z Generating XML reports...
2022-12-01T11:08:36.3445289Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105519.xml
2022-12-01T11:08:36.3445659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3445836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3446200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3446401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3446420Z 
2022-12-01T11:08:36.3446530Z Running tests...
2022-12-01T11:08:36.3446797Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3447111Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3447511Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3447532Z 
2022-12-01T11:08:36.3447794Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3447908Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3447928Z 
2022-12-01T11:08:36.3448018Z OK (skipped=1)
2022-12-01T11:08:36.3448057Z 
2022-12-01T11:08:36.3448162Z Generating XML reports...
2022-12-01T11:08:36.3448602Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105528.xml
2022-12-01T11:08:36.3449040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3449216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3449593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3449783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3449803Z 
2022-12-01T11:08:36.3449912Z Running tests...
2022-12-01T11:08:36.3450175Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3450466Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3450857Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3450880Z 
2022-12-01T11:08:36.3451140Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3451259Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3451278Z 
2022-12-01T11:08:36.3451386Z OK (skipped=1)
2022-12-01T11:08:36.3451405Z 
2022-12-01T11:08:36.3451527Z Generating XML reports...
2022-12-01T11:08:36.3451977Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105530.xml
2022-12-01T11:08:36.3452346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3452520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3452884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3453075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3453095Z 
2022-12-01T11:08:36.3453202Z Running tests...
2022-12-01T11:08:36.3453471Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3453829Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3454091Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-12-01T11:08:36.3454112Z 
2022-12-01T11:08:36.3454380Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3454490Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3454509Z 
2022-12-01T11:08:36.3454617Z OK (skipped=1)
2022-12-01T11:08:36.3454636Z 
2022-12-01T11:08:36.3454741Z Generating XML reports...
2022-12-01T11:08:36.3455187Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105532.xml
2022-12-01T11:08:36.3455562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3455741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3456123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3456316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3456336Z 
2022-12-01T11:08:36.3456443Z Running tests...
2022-12-01T11:08:36.3456708Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3457002Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3457263Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI with timeout (0.002s)
2022-12-01T11:08:36.3457283Z 
2022-12-01T11:08:36.3457543Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3457655Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3457674Z 
2022-12-01T11:08:36.3457779Z OK (skipped=1)
2022-12-01T11:08:36.3457864Z 
2022-12-01T11:08:36.3457998Z Generating XML reports...
2022-12-01T11:08:36.3458450Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105535.xml
2022-12-01T11:08:36.3458829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3459005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3459369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3459561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3459581Z 
2022-12-01T11:08:36.3459689Z Running tests...
2022-12-01T11:08:36.3459950Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3460262Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3460534Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-12-01T11:08:36.3460554Z 
2022-12-01T11:08:36.3460815Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3460928Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3460947Z 
2022-12-01T11:08:36.3461054Z OK (skipped=1)
2022-12-01T11:08:36.3461073Z 
2022-12-01T11:08:36.3461177Z Generating XML reports...
2022-12-01T11:08:36.3461626Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105537.xml
2022-12-01T11:08:36.3461997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3462174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3462553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3462751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3462822Z 
2022-12-01T11:08:36.3462941Z Running tests...
2022-12-01T11:08:36.3463213Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3463510Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3463779Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3464003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81261
2022-12-01T11:08:36.3464221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81262
2022-12-01T11:08:36.3464592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3464770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3465147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3465347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3465710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3465866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3466243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3466435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3466678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3466925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3467327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3467796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3468028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3468258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3468400Z skip: Skipped due to small world size. (3.944s)
2022-12-01T11:08:36.3468419Z 
2022-12-01T11:08:36.3468689Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3468802Z Ran 1 test in 3.944s
2022-12-01T11:08:36.3468822Z 
2022-12-01T11:08:36.3468928Z OK (skipped=1)
2022-12-01T11:08:36.3468946Z 
2022-12-01T11:08:36.3469069Z Generating XML reports...
2022-12-01T11:08:36.3469756Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105539.xml
2022-12-01T11:08:36.3470156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3470333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3470733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3470925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3470946Z 
2022-12-01T11:08:36.3471055Z Running tests...
2022-12-01T11:08:36.3471324Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3471637Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3471896Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support CPU barrier (0.002s)
2022-12-01T11:08:36.3471916Z 
2022-12-01T11:08:36.3472178Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3472297Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3472390Z 
2022-12-01T11:08:36.3472508Z OK (skipped=1)
2022-12-01T11:08:36.3472527Z 
2022-12-01T11:08:36.3472631Z Generating XML reports...
2022-12-01T11:08:36.3473082Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105545.xml
2022-12-01T11:08:36.3473455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3473633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3474012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3474205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3474225Z 
2022-12-01T11:08:36.3474330Z Running tests...
2022-12-01T11:08:36.3474597Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3474917Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3475162Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3475385Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81628
2022-12-01T11:08:36.3475603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81629
2022-12-01T11:08:36.3475977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3476156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3476536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3476728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3477176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3477333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3477709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3477896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3478142Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3478386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3478788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3479190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3479430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3479660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3479803Z skip: Skipped due to small world size. (4.016s)
2022-12-01T11:08:36.3479824Z 
2022-12-01T11:08:36.3480100Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3480210Z Ran 1 test in 4.016s
2022-12-01T11:08:36.3480229Z 
2022-12-01T11:08:36.3480337Z OK (skipped=1)
2022-12-01T11:08:36.3480356Z 
2022-12-01T11:08:36.3480478Z Generating XML reports...
2022-12-01T11:08:36.3480929Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105547.xml
2022-12-01T11:08:36.3481302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3481477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3481950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3482134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3482154Z 
2022-12-01T11:08:36.3482266Z Running tests...
2022-12-01T11:08:36.3482535Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3482846Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3483129Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T11:08:36.3483149Z 
2022-12-01T11:08:36.3483408Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3483521Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3483540Z 
2022-12-01T11:08:36.3483649Z OK (skipped=1)
2022-12-01T11:08:36.3483668Z 
2022-12-01T11:08:36.3483780Z Generating XML reports...
2022-12-01T11:08:36.3484229Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105553.xml
2022-12-01T11:08:36.3484602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3484777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3485157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3485348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3485368Z 
2022-12-01T11:08:36.3485476Z Running tests...
2022-12-01T11:08:36.3485741Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3486055Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3486314Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T11:08:36.3486386Z 
2022-12-01T11:08:36.3486661Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3486778Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3486798Z 
2022-12-01T11:08:36.3486906Z OK (skipped=1)
2022-12-01T11:08:36.3486926Z 
2022-12-01T11:08:36.3487048Z Generating XML reports...
2022-12-01T11:08:36.3487493Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105556.xml
2022-12-01T11:08:36.3487864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3488042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3488426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3488604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3488623Z 
2022-12-01T11:08:36.3488735Z Running tests...
2022-12-01T11:08:36.3489001Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3489316Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3489592Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T11:08:36.3489612Z 
2022-12-01T11:08:36.3489876Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3489989Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3490008Z 
2022-12-01T11:08:36.3490114Z OK (skipped=1)
2022-12-01T11:08:36.3490133Z 
2022-12-01T11:08:36.3490237Z Generating XML reports...
2022-12-01T11:08:36.3490680Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105558.xml
2022-12-01T11:08:36.3491112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3491296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3491679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3491873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3491893Z 
2022-12-01T11:08:36.3492002Z Running tests...
2022-12-01T11:08:36.3492270Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3492581Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3492819Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T11:08:36.3492858Z 
2022-12-01T11:08:36.3493101Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3493216Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3493235Z 
2022-12-01T11:08:36.3493344Z OK (skipped=1)
2022-12-01T11:08:36.3493363Z 
2022-12-01T11:08:36.3493487Z Generating XML reports...
2022-12-01T11:08:36.3493931Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105600.xml
2022-12-01T11:08:36.3494303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3494480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3494861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3495033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3495052Z 
2022-12-01T11:08:36.3495161Z Running tests...
2022-12-01T11:08:36.3495427Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3495808Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3496073Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T11:08:36.3496093Z 
2022-12-01T11:08:36.3496355Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3496468Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3496487Z 
2022-12-01T11:08:36.3496593Z OK (skipped=1)
2022-12-01T11:08:36.3496612Z 
2022-12-01T11:08:36.3496736Z Generating XML reports...
2022-12-01T11:08:36.3497161Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105602.xml
2022-12-01T11:08:36.3497528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3497705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3498092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3498284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3498304Z 
2022-12-01T11:08:36.3498411Z Running tests...
2022-12-01T11:08:36.3498670Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3498980Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3499235Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:08:36.3499274Z 
2022-12-01T11:08:36.3499519Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3499631Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3499650Z 
2022-12-01T11:08:36.3499758Z OK (skipped=1)
2022-12-01T11:08:36.3499776Z 
2022-12-01T11:08:36.3499902Z Generating XML reports...
2022-12-01T11:08:36.3500400Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105604.xml
2022-12-01T11:08:36.3500786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3500961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3501343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3501517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3501555Z 
2022-12-01T11:08:36.3501644Z Running tests...
2022-12-01T11:08:36.3501903Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3502210Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3502465Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-12-01T11:08:36.3502489Z 
2022-12-01T11:08:36.3502754Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3502865Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3502884Z 
2022-12-01T11:08:36.3502990Z OK (skipped=1)
2022-12-01T11:08:36.3503009Z 
2022-12-01T11:08:36.3503133Z Generating XML reports...
2022-12-01T11:08:36.3503558Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105606.xml
2022-12-01T11:08:36.3503929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3504105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3504488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3504680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3504761Z 
2022-12-01T11:08:36.3504877Z Running tests...
2022-12-01T11:08:36.3505147Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3505457Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3505710Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:08:36.3505749Z 
2022-12-01T11:08:36.3505990Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3506105Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3506124Z 
2022-12-01T11:08:36.3506229Z OK (skipped=1)
2022-12-01T11:08:36.3506248Z 
2022-12-01T11:08:36.3506368Z Generating XML reports...
2022-12-01T11:08:36.3506818Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105608.xml
2022-12-01T11:08:36.3507188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3507370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3507753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3507926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3507964Z 
2022-12-01T11:08:36.3508053Z Running tests...
2022-12-01T11:08:36.3508318Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3508709Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3508970Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:08:36.3508989Z 
2022-12-01T11:08:36.3509252Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3509607Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3509629Z 
2022-12-01T11:08:36.3509745Z OK (skipped=1)
2022-12-01T11:08:36.3509838Z 
2022-12-01T11:08:36.3509972Z Generating XML reports...
2022-12-01T11:08:36.3510417Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105610.xml
2022-12-01T11:08:36.3510791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3511082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3511461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3511654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3511674Z 
2022-12-01T11:08:36.3511783Z Running tests...
2022-12-01T11:08:36.3512048Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3512366Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3512637Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:08:36.3512657Z 
2022-12-01T11:08:36.3512902Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3513015Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3513035Z 
2022-12-01T11:08:36.3513142Z OK (skipped=1)
2022-12-01T11:08:36.3513162Z 
2022-12-01T11:08:36.3513285Z Generating XML reports...
2022-12-01T11:08:36.3513725Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105612.xml
2022-12-01T11:08:36.3514095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3514271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3514651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3514903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3514942Z 
2022-12-01T11:08:36.3515033Z Running tests...
2022-12-01T11:08:36.3515300Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3515609Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3515886Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-12-01T11:08:36.3515906Z 
2022-12-01T11:08:36.3516163Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3516275Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3516294Z 
2022-12-01T11:08:36.3516402Z OK (skipped=1)
2022-12-01T11:08:36.3516421Z 
2022-12-01T11:08:36.3516547Z Generating XML reports...
2022-12-01T11:08:36.3516976Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105614.xml
2022-12-01T11:08:36.3517358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3517533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3517915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3518108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3518128Z 
2022-12-01T11:08:36.3518238Z Running tests...
2022-12-01T11:08:36.3518502Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3518811Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3519073Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:08:36.3519097Z 
2022-12-01T11:08:36.3519390Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3519514Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3519534Z 
2022-12-01T11:08:36.3519653Z OK (skipped=1)
2022-12-01T11:08:36.3519671Z 
2022-12-01T11:08:36.3519818Z Generating XML reports...
2022-12-01T11:08:36.3520267Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105616.xml
2022-12-01T11:08:36.3520638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3520810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3521193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3521389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3521415Z 
2022-12-01T11:08:36.3521525Z Running tests...
2022-12-01T11:08:36.3521775Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3522087Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3522356Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:08:36.3522375Z 
2022-12-01T11:08:36.3522635Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3522746Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3522765Z 
2022-12-01T11:08:36.3522871Z OK (skipped=1)
2022-12-01T11:08:36.3522890Z 
2022-12-01T11:08:36.3523009Z Generating XML reports...
2022-12-01T11:08:36.3523452Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105618.xml
2022-12-01T11:08:36.3523883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3524122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3524510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3524701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3524720Z 
2022-12-01T11:08:36.3524827Z Running tests...
2022-12-01T11:08:36.3525094Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3525402Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3525648Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3525871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83188
2022-12-01T11:08:36.3526072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83189
2022-12-01T11:08:36.3526452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3526631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3527008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3527199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3527561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3527734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3528106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3528296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3528526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3528831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3529250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3529647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3529879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3530218Z STAGE:2022-12-01 10:56:24 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3530442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3530767Z STAGE:2022-12-01 10:56:24 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3531013Z [1669892186.391589] [7ff5451e42d4:83188:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3531237Z [1669892186.391589] [7ff5451e42d4:83188:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3531468Z [1669892186.415721] [7ff5451e42d4:83189:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3531705Z [1669892186.415721] [7ff5451e42d4:83189:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3532258Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3532279Z 
2022-12-01T11:08:36.3532606Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3533011Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3533336Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3533657Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3533989Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3534320Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3534638Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3534968Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3535294Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3535625Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3535960Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3536290Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3536618Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3536939Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3537269Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3537588Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3537914Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3538297Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3538640Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3538968Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3539293Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3539615Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3539948Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3540260Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3540581Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3540913Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3541247Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3541576Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3541900Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3542218Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3542550Z STAGE:2022-12-01 10:56:26 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3542877Z STAGE:2022-12-01 10:56:26 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3543186Z STAGE:2022-12-01 10:56:27 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3543576Z STAGE:2022-12-01 10:56:27 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3543903Z STAGE:2022-12-01 10:56:27 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3544235Z STAGE:2022-12-01 10:56:27 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3544561Z STAGE:2022-12-01 10:56:27 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3544881Z STAGE:2022-12-01 10:56:27 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3545215Z STAGE:2022-12-01 10:56:27 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3545545Z STAGE:2022-12-01 10:56:27 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3545869Z STAGE:2022-12-01 10:56:27 83188:83188 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3546182Z STAGE:2022-12-01 10:56:27 83189:83189 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3546518Z STAGE:2022-12-01 10:56:27 83188:83188 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3546843Z STAGE:2022-12-01 10:56:27 83189:83189 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3546948Z ok (7.187s)
2022-12-01T11:08:36.3546968Z 
2022-12-01T11:08:36.3547236Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3547346Z Ran 1 test in 7.187s
2022-12-01T11:08:36.3547367Z 
2022-12-01T11:08:36.3547460Z OK
2022-12-01T11:08:36.3547481Z 
2022-12-01T11:08:36.3547603Z Generating XML reports...
2022-12-01T11:08:36.3548054Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105620.xml
2022-12-01T11:08:36.3548409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3548643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3549044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3549237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3549257Z 
2022-12-01T11:08:36.3549606Z Running tests...
2022-12-01T11:08:36.3549891Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3550206Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3550493Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo and Nccl backend supports CUDA allReduce (0.002s)
2022-12-01T11:08:36.3550513Z 
2022-12-01T11:08:36.3550777Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3550876Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3550896Z 
2022-12-01T11:08:36.3551002Z OK (skipped=1)
2022-12-01T11:08:36.3551025Z 
2022-12-01T11:08:36.3551148Z Generating XML reports...
2022-12-01T11:08:36.3551599Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105630.xml
2022-12-01T11:08:36.3551974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3552148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3552532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3552725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3552745Z 
2022-12-01T11:08:36.3552834Z Running tests...
2022-12-01T11:08:36.3553100Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3553512Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3553781Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3554004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83653
2022-12-01T11:08:36.3554223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83654
2022-12-01T11:08:36.3554602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3554779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3555160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3555333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3555700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3555882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3556257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3556444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3556693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3556937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3557340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3557720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3557955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3558262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3558501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3558742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3559143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3559539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3559877Z STAGE:2022-12-01 10:56:36 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3560209Z STAGE:2022-12-01 10:56:36 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3560437Z [1669892197.827112] [7ff5451e42d4:83654:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3560680Z [1669892197.827112] [7ff5451e42d4:83654:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3561024Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3561257Z [1669892197.851482] [7ff5451e42d4:83653:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3561495Z [1669892197.851482] [7ff5451e42d4:83653:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3561837Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3562170Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3562575Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3562909Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3563231Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3563545Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3563866Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3564204Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3564533Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3564861Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3565189Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3565521Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3565859Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3566191Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3566499Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3566830Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3567158Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3567485Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3567861Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3568214Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3568542Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3568865Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3569192Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3569509Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3569835Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3570158Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3570486Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3570879Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3571211Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3571537Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3571856Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3572166Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3572497Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3572827Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3573217Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3573545Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3573873Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3574199Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3592002Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3592473Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3592823Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3593140Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3593495Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3593835Z STAGE:2022-12-01 10:56:38 83653:83653 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3594167Z STAGE:2022-12-01 10:56:38 83654:83654 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3594269Z ok (7.196s)
2022-12-01T11:08:36.3594292Z 
2022-12-01T11:08:36.3594567Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3594684Z Ran 1 test in 7.196s
2022-12-01T11:08:36.3594704Z 
2022-12-01T11:08:36.3594797Z OK
2022-12-01T11:08:36.3594816Z 
2022-12-01T11:08:36.3594922Z Generating XML reports...
2022-12-01T11:08:36.3595378Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105632.xml
2022-12-01T11:08:36.3595757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3596102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3596512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3596708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3596728Z 
2022-12-01T11:08:36.3596838Z Running tests...
2022-12-01T11:08:36.3597111Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3597430Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3597675Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3597897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84070
2022-12-01T11:08:36.3598119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84071
2022-12-01T11:08:36.3598511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3598690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3599077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3599270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3599640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3599797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3600171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3600360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3600701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3600953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3601359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3601761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3601993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3602223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3602364Z skip: Skipped due to small world size. (3.967s)
2022-12-01T11:08:36.3602385Z 
2022-12-01T11:08:36.3602654Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3602774Z Ran 1 test in 3.967s
2022-12-01T11:08:36.3602793Z 
2022-12-01T11:08:36.3602904Z OK (skipped=1)
2022-12-01T11:08:36.3602923Z 
2022-12-01T11:08:36.3603049Z Generating XML reports...
2022-12-01T11:08:36.3603504Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105641.xml
2022-12-01T11:08:36.3603881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3604057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3604439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3604615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3604634Z 
2022-12-01T11:08:36.3604744Z Running tests...
2022-12-01T11:08:36.3605009Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3605377Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3605648Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3605868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84311
2022-12-01T11:08:36.3606085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84312
2022-12-01T11:08:36.3606460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3606619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3607001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3607195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3607559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3607743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3608127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3608318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3608566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3608817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3609207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3609607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3609839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3610151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3610941Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:08:36.3611056Z   warnings.warn(
2022-12-01T11:08:36.3611824Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:08:36.3611934Z   warnings.warn(
2022-12-01T11:08:36.3612170Z [1669892213.246512] [7ff5451e42d4:84312:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3612420Z [1669892213.246512] [7ff5451e42d4:84312:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3612635Z [1669892213.244881] [7ff5451e42d4:84311:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3612863Z [1669892213.244881] [7ff5451e42d4:84311:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3612966Z ok (6.024s)
2022-12-01T11:08:36.3612986Z 
2022-12-01T11:08:36.3613259Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3613371Z Ran 1 test in 6.024s
2022-12-01T11:08:36.3613391Z 
2022-12-01T11:08:36.3613483Z OK
2022-12-01T11:08:36.3613502Z 
2022-12-01T11:08:36.3613625Z Generating XML reports...
2022-12-01T11:08:36.3614075Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105648.xml
2022-12-01T11:08:36.3614508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3614674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3615061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3615255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3615274Z 
2022-12-01T11:08:36.3615381Z Running tests...
2022-12-01T11:08:36.3615646Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3615960Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3616370Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3616394Z 
2022-12-01T11:08:36.3616667Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3616762Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3616801Z 
2022-12-01T11:08:36.3616890Z OK (skipped=1)
2022-12-01T11:08:36.3616908Z 
2022-12-01T11:08:36.3617035Z Generating XML reports...
2022-12-01T11:08:36.3617485Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105656.xml
2022-12-01T11:08:36.3617857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3618036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3618420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3618610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3618631Z 
2022-12-01T11:08:36.3618797Z Running tests...
2022-12-01T11:08:36.3619049Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3619362Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3619844Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3619865Z 
2022-12-01T11:08:36.3620126Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3620239Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3620258Z 
2022-12-01T11:08:36.3620367Z OK (skipped=1)
2022-12-01T11:08:36.3620386Z 
2022-12-01T11:08:36.3620510Z Generating XML reports...
2022-12-01T11:08:36.3620957Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105658.xml
2022-12-01T11:08:36.3621329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3621494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3621875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3622069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3622088Z 
2022-12-01T11:08:36.3622197Z Running tests...
2022-12-01T11:08:36.3622463Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3622778Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3623268Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3623288Z 
2022-12-01T11:08:36.3623553Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3623669Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3623689Z 
2022-12-01T11:08:36.3623829Z OK (skipped=1)
2022-12-01T11:08:36.3623850Z 
2022-12-01T11:08:36.3623982Z Generating XML reports...
2022-12-01T11:08:36.3624430Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105700.xml
2022-12-01T11:08:36.3624804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3624980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3625360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3625552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3625572Z 
2022-12-01T11:08:36.3625683Z Running tests...
2022-12-01T11:08:36.3625949Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3626258Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3626523Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3626744Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85020
2022-12-01T11:08:36.3626962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85021
2022-12-01T11:08:36.3627332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3627508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3627888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3628082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3628433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3628672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3629054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3629244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3629742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3629998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3630412Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3630815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3631052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3631268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3631533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsas5oph8
2022-12-01T11:08:36.3631807Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsas5oph8/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3632063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbd4djh9a
2022-12-01T11:08:36.3632333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbd4djh9a/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3632568Z [1669892227.753421] [7ff5451e42d4:85020:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3632809Z [1669892227.753421] [7ff5451e42d4:85020:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3633129Z [1669892227.756185] [7ff5451e42d4:85021:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3633383Z [1669892227.756185] [7ff5451e42d4:85021:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3633603Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3633847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3633949Z ok (6.452s)
2022-12-01T11:08:36.3633970Z 
2022-12-01T11:08:36.3634244Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3634358Z Ran 1 test in 6.452s
2022-12-01T11:08:36.3634378Z 
2022-12-01T11:08:36.3634471Z OK
2022-12-01T11:08:36.3634490Z 
2022-12-01T11:08:36.3634614Z Generating XML reports...
2022-12-01T11:08:36.3635068Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105702.xml
2022-12-01T11:08:36.3635455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3635613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3635999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3636191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3636211Z 
2022-12-01T11:08:36.3636319Z Running tests...
2022-12-01T11:08:36.3636584Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3636899Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3637180Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3637481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85459
2022-12-01T11:08:36.3637685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85460
2022-12-01T11:08:36.3638067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3638243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3638627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3638821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3639191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3639367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3639745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3639946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3640174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3640419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3640825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3641226Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3641454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3641682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3641939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxj1_xmba
2022-12-01T11:08:36.3642264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxj1_xmba/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3642508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3juemau7
2022-12-01T11:08:36.3642779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3juemau7/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3643019Z [1669892236.529930] [7ff5451e42d4:85460:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3643262Z [1669892236.529930] [7ff5451e42d4:85460:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3643491Z [1669892236.522642] [7ff5451e42d4:85459:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3643732Z [1669892236.522642] [7ff5451e42d4:85459:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3643978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3644216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3644445Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3644678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3644761Z ok (6.612s)
2022-12-01T11:08:36.3644782Z 
2022-12-01T11:08:36.3645061Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3645173Z Ran 1 test in 6.612s
2022-12-01T11:08:36.3645193Z 
2022-12-01T11:08:36.3645286Z OK
2022-12-01T11:08:36.3645305Z 
2022-12-01T11:08:36.3645429Z Generating XML reports...
2022-12-01T11:08:36.3645882Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105711.xml
2022-12-01T11:08:36.3646359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3646546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3648736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3648934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3648954Z 
2022-12-01T11:08:36.3649065Z Running tests...
2022-12-01T11:08:36.3649331Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3649645Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3649919Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3650683Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.692s)
2022-12-01T11:08:36.3650708Z 
2022-12-01T11:08:36.3650978Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3651093Z Ran 1 test in 1.692s
2022-12-01T11:08:36.3651112Z 
2022-12-01T11:08:36.3651220Z OK (skipped=1)
2022-12-01T11:08:36.3651239Z 
2022-12-01T11:08:36.3651345Z Generating XML reports...
2022-12-01T11:08:36.3651799Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105720.xml
2022-12-01T11:08:36.3652175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3652354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3652737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3652988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3653010Z 
2022-12-01T11:08:36.3653126Z Running tests...
2022-12-01T11:08:36.3653396Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3653712Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3653986Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3654733Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.598s)
2022-12-01T11:08:36.3654757Z 
2022-12-01T11:08:36.3655023Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3655140Z Ran 1 test in 1.599s
2022-12-01T11:08:36.3655159Z 
2022-12-01T11:08:36.3655267Z OK (skipped=1)
2022-12-01T11:08:36.3655285Z 
2022-12-01T11:08:36.3655410Z Generating XML reports...
2022-12-01T11:08:36.3655856Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105724.xml
2022-12-01T11:08:36.3656233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3656413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3656777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3656971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3656990Z 
2022-12-01T11:08:36.3657099Z Running tests...
2022-12-01T11:08:36.3657421Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3657741Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3658182Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T11:08:36.3658202Z 
2022-12-01T11:08:36.3658463Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3658578Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3658597Z 
2022-12-01T11:08:36.3658706Z OK (skipped=1)
2022-12-01T11:08:36.3658725Z 
2022-12-01T11:08:36.3658830Z Generating XML reports...
2022-12-01T11:08:36.3659281Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105728.xml
2022-12-01T11:08:36.3659657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3659842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3660224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3660418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3660438Z 
2022-12-01T11:08:36.3660547Z Running tests...
2022-12-01T11:08:36.3660812Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3661125Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3661414Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3661641Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86231
2022-12-01T11:08:36.3661860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86232
2022-12-01T11:08:36.3662298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3662487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3662876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3663070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3663438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3663595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3663969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3664158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3664413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3664663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3665067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3665468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3665703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3665930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3666171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5j215qbd
2022-12-01T11:08:36.3666443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5j215qbd/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3666759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1n44d1zo
2022-12-01T11:08:36.3667028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1n44d1zo/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3667266Z [1669892255.337351] [7ff5451e42d4:86231:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3667510Z [1669892255.337351] [7ff5451e42d4:86231:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3667736Z [1669892255.343745] [7ff5451e42d4:86232:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3667967Z [1669892255.343745] [7ff5451e42d4:86232:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3668071Z ok (6.006s)
2022-12-01T11:08:36.3668090Z 
2022-12-01T11:08:36.3668365Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3668467Z Ran 1 test in 6.006s
2022-12-01T11:08:36.3668488Z 
2022-12-01T11:08:36.3668581Z OK
2022-12-01T11:08:36.3668601Z 
2022-12-01T11:08:36.3668727Z Generating XML reports...
2022-12-01T11:08:36.3669181Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105730.xml
2022-12-01T11:08:36.3669841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3670029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3670426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3670622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3670675Z 
2022-12-01T11:08:36.3670770Z Running tests...
2022-12-01T11:08:36.3671040Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3671440Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3671718Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3671939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86601
2022-12-01T11:08:36.3672158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86602
2022-12-01T11:08:36.3672541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3672717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3673081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3673277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3673652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3673828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3674204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3674399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3674648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3674893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3675300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3675684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3676009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3676241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3676503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg72q5rqq
2022-12-01T11:08:36.3676775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg72q5rqq/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3677033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp34oa1t_
2022-12-01T11:08:36.3677300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp34oa1t_/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3677538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3677777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3677995Z [1669892263.666683] [7ff5451e42d4:86601:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3678246Z [1669892263.666683] [7ff5451e42d4:86601:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3678479Z [1669892263.670761] [7ff5451e42d4:86602:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3678721Z [1669892263.670761] [7ff5451e42d4:86602:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3678826Z ok (6.635s)
2022-12-01T11:08:36.3678846Z 
2022-12-01T11:08:36.3679124Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3679239Z Ran 1 test in 6.636s
2022-12-01T11:08:36.3679258Z 
2022-12-01T11:08:36.3679351Z OK
2022-12-01T11:08:36.3679370Z 
2022-12-01T11:08:36.3679494Z Generating XML reports...
2022-12-01T11:08:36.3679927Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105738.xml
2022-12-01T11:08:36.3680355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3680542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3680926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3681118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3681138Z 
2022-12-01T11:08:36.3681248Z Running tests...
2022-12-01T11:08:36.3681514Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3681829Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3682251Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.004s)
2022-12-01T11:08:36.3682295Z 
2022-12-01T11:08:36.3682544Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3682656Z Ran 1 test in 0.004s
2022-12-01T11:08:36.3682675Z 
2022-12-01T11:08:36.3682784Z OK (skipped=1)
2022-12-01T11:08:36.3682802Z 
2022-12-01T11:08:36.3682926Z Generating XML reports...
2022-12-01T11:08:36.3683376Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105747.xml
2022-12-01T11:08:36.3683754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3683933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3684316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3684493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3684582Z 
2022-12-01T11:08:36.3684677Z Running tests...
2022-12-01T11:08:36.3684951Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3685264Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3685696Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.007s)
2022-12-01T11:08:36.3685715Z 
2022-12-01T11:08:36.3685976Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3686092Z Ran 1 test in 0.007s
2022-12-01T11:08:36.3686111Z 
2022-12-01T11:08:36.3686220Z OK (skipped=1)
2022-12-01T11:08:36.3686238Z 
2022-12-01T11:08:36.3686361Z Generating XML reports...
2022-12-01T11:08:36.3686790Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105749.xml
2022-12-01T11:08:36.3687164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3687350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3687734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3687929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3687948Z 
2022-12-01T11:08:36.3688058Z Running tests...
2022-12-01T11:08:36.3688321Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3688636Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3688896Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3689099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87206
2022-12-01T11:08:36.3689318Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87207
2022-12-01T11:08:36.3689746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3689932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3690321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3690514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3690882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3691059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3691416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3691606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3691863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3692109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3692515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3692918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3693149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3693408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wzjhpd9
2022-12-01T11:08:36.3693681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wzjhpd9/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3693891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3694205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkkl2jdx
2022-12-01T11:08:36.3694475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkkl2jdx/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3694713Z [1669892276.906034] [7ff5451e42d4:87206:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3694957Z [1669892276.906034] [7ff5451e42d4:87206:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3695866Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3696113Z [1669892276.916504] [7ff5451e42d4:87207:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3696357Z [1669892276.916504] [7ff5451e42d4:87207:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3697242Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3698451Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T11:08:36.3698699Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T11:08:36.3699863Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T11:08:36.3700097Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T11:08:36.3700344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3700583Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3701470Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3702351Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3703300Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3704179Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3705066Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3705945Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3706817Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3707733Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3708623Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3709873Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:08:36.3710003Z ok (6.235s)
2022-12-01T11:08:36.3710025Z 
2022-12-01T11:08:36.3710309Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3710404Z Ran 1 test in 6.235s
2022-12-01T11:08:36.3710425Z 
2022-12-01T11:08:36.3710519Z OK
2022-12-01T11:08:36.3710537Z 
2022-12-01T11:08:36.3710663Z Generating XML reports...
2022-12-01T11:08:36.3711118Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105751.xml
2022-12-01T11:08:36.3711493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3711671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3712057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3712346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3712369Z 
2022-12-01T11:08:36.3712460Z Running tests...
2022-12-01T11:08:36.3712735Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3713052Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3713442Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.005s)
2022-12-01T11:08:36.3713462Z 
2022-12-01T11:08:36.3713726Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3713839Z Ran 1 test in 0.005s
2022-12-01T11:08:36.3713858Z 
2022-12-01T11:08:36.3713966Z OK (skipped=1)
2022-12-01T11:08:36.3713985Z 
2022-12-01T11:08:36.3714116Z Generating XML reports...
2022-12-01T11:08:36.3714570Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105800.xml
2022-12-01T11:08:36.3714934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3715115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3715501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3715696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3715716Z 
2022-12-01T11:08:36.3715825Z Running tests...
2022-12-01T11:08:36.3716090Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3716405Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3716820Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T11:08:36.3716842Z 
2022-12-01T11:08:36.3717111Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3717265Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3717286Z 
2022-12-01T11:08:36.3717402Z OK (skipped=1)
2022-12-01T11:08:36.3717420Z 
2022-12-01T11:08:36.3717547Z Generating XML reports...
2022-12-01T11:08:36.3717996Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105802.xml
2022-12-01T11:08:36.3718370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3718546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3718929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3719121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3719141Z 
2022-12-01T11:08:36.3719250Z Running tests...
2022-12-01T11:08:36.3719502Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3719819Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3720090Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3720841Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.745s)
2022-12-01T11:08:36.3720862Z 
2022-12-01T11:08:36.3721130Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3721244Z Ran 1 test in 1.745s
2022-12-01T11:08:36.3721264Z 
2022-12-01T11:08:36.3721373Z OK (skipped=1)
2022-12-01T11:08:36.3721392Z 
2022-12-01T11:08:36.3721518Z Generating XML reports...
2022-12-01T11:08:36.3722037Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105804.xml
2022-12-01T11:08:36.3722396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3722577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3722962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3723157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3723176Z 
2022-12-01T11:08:36.3723286Z Running tests...
2022-12-01T11:08:36.3723550Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3723863Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3724137Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3724889Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2022-12-01T11:08:36.3724910Z 
2022-12-01T11:08:36.3725177Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3725270Z Ran 1 test in 1.631s
2022-12-01T11:08:36.3725289Z 
2022-12-01T11:08:36.3725399Z OK (skipped=1)
2022-12-01T11:08:36.3725418Z 
2022-12-01T11:08:36.3725543Z Generating XML reports...
2022-12-01T11:08:36.3725990Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105808.xml
2022-12-01T11:08:36.3726366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3726550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3726987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3727191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3727210Z 
2022-12-01T11:08:36.3727318Z Running tests...
2022-12-01T11:08:36.3727569Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3727879Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3728170Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3728392Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88092
2022-12-01T11:08:36.3728612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88093
2022-12-01T11:08:36.3728993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3729173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3729560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3729734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3730101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3730276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3730653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3730845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3731095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3731415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3731823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3732226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3732441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3732687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3732914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3733153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3733568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3733965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3734228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4afe4mrt
2022-12-01T11:08:36.3734501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4afe4mrt/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3734758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19aepsxg
2022-12-01T11:08:36.3735008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19aepsxg/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3735245Z [1669892297.378177] [7ff5451e42d4:88093:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3735487Z [1669892297.378177] [7ff5451e42d4:88093:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3735769Z [1669892297.374303] [7ff5451e42d4:88092:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3736014Z [1669892297.374303] [7ff5451e42d4:88092:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3736255Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3736492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3736728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3736961Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3737045Z ok (6.703s)
2022-12-01T11:08:36.3737065Z 
2022-12-01T11:08:36.3737346Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3737466Z Ran 1 test in 6.703s
2022-12-01T11:08:36.3737487Z 
2022-12-01T11:08:36.3737583Z OK
2022-12-01T11:08:36.3737602Z 
2022-12-01T11:08:36.3737727Z Generating XML reports...
2022-12-01T11:08:36.3738181Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105812.xml
2022-12-01T11:08:36.3738555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3738736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3739101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3739296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3739316Z 
2022-12-01T11:08:36.3739426Z Running tests...
2022-12-01T11:08:36.3739695Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3740091Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3740371Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3740594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88420
2022-12-01T11:08:36.3740814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88421
2022-12-01T11:08:36.3741189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3741347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3741725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3741918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3742284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3742467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3742841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3743030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3743278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3743504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3743909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3744309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3744545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3744873Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:08:36.3745111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3745389Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:08:36.3745647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd53101a
2022-12-01T11:08:36.3745920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd53101a/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3746158Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8z6qqahg
2022-12-01T11:08:36.3746434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8z6qqahg/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3746680Z [1669892306.261721] [7ff5451e42d4:88421:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3746927Z [1669892306.261721] [7ff5451e42d4:88421:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3747155Z [1669892306.260752] [7ff5451e42d4:88420:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3747393Z [1669892306.260752] [7ff5451e42d4:88420:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3747632Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3747870Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3748105Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3748339Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3748656Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:08:36.3748937Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:08:36.3749213Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:08:36.3749743Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:08:36.3749989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3750224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3750460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3750699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3750961Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:08:36.3751237Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:08:36.3751514Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T11:08:36.3751783Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T11:08:36.3752018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3752253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3752477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3752713Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3752889Z ok (7.152s)
2022-12-01T11:08:36.3752911Z 
2022-12-01T11:08:36.3753191Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3753306Z Ran 1 test in 7.153s
2022-12-01T11:08:36.3753326Z 
2022-12-01T11:08:36.3753421Z OK
2022-12-01T11:08:36.3753440Z 
2022-12-01T11:08:36.3753565Z Generating XML reports...
2022-12-01T11:08:36.3754020Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105821.xml
2022-12-01T11:08:36.3754400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3754582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3754966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3755165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3755188Z 
2022-12-01T11:08:36.3755279Z Running tests...
2022-12-01T11:08:36.3755552Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3755869Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3756143Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3756894Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.708s)
2022-12-01T11:08:36.3756915Z 
2022-12-01T11:08:36.3757185Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3757378Z Ran 1 test in 1.709s
2022-12-01T11:08:36.3757398Z 
2022-12-01T11:08:36.3757507Z OK (skipped=1)
2022-12-01T11:08:36.3757528Z 
2022-12-01T11:08:36.3757655Z Generating XML reports...
2022-12-01T11:08:36.3758084Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105830.xml
2022-12-01T11:08:36.3758463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3758642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3759028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3759222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3759242Z 
2022-12-01T11:08:36.3759352Z Running tests...
2022-12-01T11:08:36.3759620Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3759934Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3760400Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... skip: flaky on PyTorch CI: No such file or directory: '/tmp/checkpoint.pt' (0.002s)
2022-12-01T11:08:36.3760421Z 
2022-12-01T11:08:36.3760664Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3760780Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3760798Z 
2022-12-01T11:08:36.3760909Z OK (skipped=1)
2022-12-01T11:08:36.3760928Z 
2022-12-01T11:08:36.3761053Z Generating XML reports...
2022-12-01T11:08:36.3761504Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105834.xml
2022-12-01T11:08:36.3761882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3762058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3762496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3762701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3762720Z 
2022-12-01T11:08:36.3762810Z Running tests...
2022-12-01T11:08:36.3763080Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3763396Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3763791Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3763811Z 
2022-12-01T11:08:36.3764077Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3764191Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3764210Z 
2022-12-01T11:08:36.3764318Z OK (skipped=1)
2022-12-01T11:08:36.3764341Z 
2022-12-01T11:08:36.3764466Z Generating XML reports...
2022-12-01T11:08:36.3764920Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105836.xml
2022-12-01T11:08:36.3765278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3765457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3765839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3766034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3766053Z 
2022-12-01T11:08:36.3766164Z Running tests...
2022-12-01T11:08:36.3766433Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3766751Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3767208Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3767228Z 
2022-12-01T11:08:36.3767495Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3767587Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3767606Z 
2022-12-01T11:08:36.3767719Z OK (skipped=1)
2022-12-01T11:08:36.3767738Z 
2022-12-01T11:08:36.3767862Z Generating XML reports...
2022-12-01T11:08:36.3768309Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105838.xml
2022-12-01T11:08:36.3768683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3768861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3769241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3769442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3769462Z 
2022-12-01T11:08:36.3769571Z Running tests...
2022-12-01T11:08:36.3769818Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3770130Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3770580Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3770600Z 
2022-12-01T11:08:36.3770916Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3771030Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3771049Z 
2022-12-01T11:08:36.3771158Z OK (skipped=1)
2022-12-01T11:08:36.3771182Z 
2022-12-01T11:08:36.3771311Z Generating XML reports...
2022-12-01T11:08:36.3771813Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105840.xml
2022-12-01T11:08:36.3772203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3772362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3772750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3772946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3772966Z 
2022-12-01T11:08:36.3773076Z Running tests...
2022-12-01T11:08:36.3773342Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3773659Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3774118Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3774138Z 
2022-12-01T11:08:36.3774409Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3774522Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3774541Z 
2022-12-01T11:08:36.3774630Z OK (skipped=1)
2022-12-01T11:08:36.3774670Z 
2022-12-01T11:08:36.3774775Z Generating XML reports...
2022-12-01T11:08:36.3775226Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105842.xml
2022-12-01T11:08:36.3775603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3775783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3776236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3776429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3776450Z 
2022-12-01T11:08:36.3776558Z Running tests...
2022-12-01T11:08:36.3776827Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3777120Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3777569Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3777590Z 
2022-12-01T11:08:36.3777854Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3777966Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3777989Z 
2022-12-01T11:08:36.3778098Z OK (skipped=1)
2022-12-01T11:08:36.3778117Z 
2022-12-01T11:08:36.3778245Z Generating XML reports...
2022-12-01T11:08:36.3778698Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105844.xml
2022-12-01T11:08:36.3779074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3779253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3779617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3779814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3779834Z 
2022-12-01T11:08:36.3779945Z Running tests...
2022-12-01T11:08:36.3780212Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3780524Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3781025Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3781048Z 
2022-12-01T11:08:36.3781323Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3781436Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3781455Z 
2022-12-01T11:08:36.3781564Z OK (skipped=1)
2022-12-01T11:08:36.3781583Z 
2022-12-01T11:08:36.3781707Z Generating XML reports...
2022-12-01T11:08:36.3782138Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105846.xml
2022-12-01T11:08:36.3782517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3782695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3783089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3783286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3783305Z 
2022-12-01T11:08:36.3783415Z Running tests...
2022-12-01T11:08:36.3783680Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3783995Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3784442Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3784462Z 
2022-12-01T11:08:36.3784707Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3784821Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3784889Z 
2022-12-01T11:08:36.3785004Z OK (skipped=1)
2022-12-01T11:08:36.3785026Z 
2022-12-01T11:08:36.3785152Z Generating XML reports...
2022-12-01T11:08:36.3785607Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105848.xml
2022-12-01T11:08:36.3785980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3786158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3786544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3786743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3786762Z 
2022-12-01T11:08:36.3786853Z Running tests...
2022-12-01T11:08:36.3787120Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3787442Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3787890Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3787910Z 
2022-12-01T11:08:36.3788176Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3788289Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3788309Z 
2022-12-01T11:08:36.3788417Z OK (skipped=1)
2022-12-01T11:08:36.3788436Z 
2022-12-01T11:08:36.3788561Z Generating XML reports...
2022-12-01T11:08:36.3789011Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105850.xml
2022-12-01T11:08:36.3789623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3789818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3790333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3790540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3790560Z 
2022-12-01T11:08:36.3790672Z Running tests...
2022-12-01T11:08:36.3790942Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3791255Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3791701Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3791722Z 
2022-12-01T11:08:36.3791995Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3792093Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3792132Z 
2022-12-01T11:08:36.3792226Z OK (skipped=1)
2022-12-01T11:08:36.3792245Z 
2022-12-01T11:08:36.3792370Z Generating XML reports...
2022-12-01T11:08:36.3792818Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105853.xml
2022-12-01T11:08:36.3793191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3793368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3793749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3793943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3793963Z 
2022-12-01T11:08:36.3794075Z Running tests...
2022-12-01T11:08:36.3794322Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3794718Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3795163Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3795183Z 
2022-12-01T11:08:36.3795450Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3795567Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3795586Z 
2022-12-01T11:08:36.3795692Z OK (skipped=1)
2022-12-01T11:08:36.3795711Z 
2022-12-01T11:08:36.3795836Z Generating XML reports...
2022-12-01T11:08:36.3796286Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105855.xml
2022-12-01T11:08:36.3796659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3796823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3797205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3797400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3797420Z 
2022-12-01T11:08:36.3797529Z Running tests...
2022-12-01T11:08:36.3797805Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3798122Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3798510Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.004s)
2022-12-01T11:08:36.3798530Z 
2022-12-01T11:08:36.3798794Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3798910Z Ran 1 test in 0.004s
2022-12-01T11:08:36.3798928Z 
2022-12-01T11:08:36.3799066Z OK (skipped=1)
2022-12-01T11:08:36.3799107Z 
2022-12-01T11:08:36.3799217Z Generating XML reports...
2022-12-01T11:08:36.3799672Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105857.xml
2022-12-01T11:08:36.3800049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3800226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3800607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3800800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3800820Z 
2022-12-01T11:08:36.3800931Z Running tests...
2022-12-01T11:08:36.3801201Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3801508Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3801897Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:08:36.3801918Z 
2022-12-01T11:08:36.3802183Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3802297Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3802315Z 
2022-12-01T11:08:36.3802424Z OK (skipped=1)
2022-12-01T11:08:36.3802443Z 
2022-12-01T11:08:36.3802566Z Generating XML reports...
2022-12-01T11:08:36.3803015Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105859.xml
2022-12-01T11:08:36.3803386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3803637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3804012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3804207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3804228Z 
2022-12-01T11:08:36.3804336Z Running tests...
2022-12-01T11:08:36.3804603Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3804916Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3805322Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3805342Z 
2022-12-01T11:08:36.3805608Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3805721Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3805740Z 
2022-12-01T11:08:36.3805848Z OK (skipped=1)
2022-12-01T11:08:36.3805871Z 
2022-12-01T11:08:36.3805976Z Generating XML reports...
2022-12-01T11:08:36.3806429Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105901.xml
2022-12-01T11:08:36.3806806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3806984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3807366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3807559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3807578Z 
2022-12-01T11:08:36.3807688Z Running tests...
2022-12-01T11:08:36.3807954Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3808266Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3808573Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3808807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90252
2022-12-01T11:08:36.3809026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90253
2022-12-01T11:08:36.3809409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3809588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3809970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3810164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3810531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3810694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3811080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3811273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3811525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3811773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3812181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3812585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3812820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3813107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3813349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8j_un7js
2022-12-01T11:08:36.3813621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8j_un7js/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3813873Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmdc1gnj4
2022-12-01T11:08:36.3814146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmdc1gnj4/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3814381Z [1669892348.601572] [7ff5451e42d4:90252:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3814627Z [1669892348.601572] [7ff5451e42d4:90252:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3814856Z [1669892348.601598] [7ff5451e42d4:90253:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3815101Z [1669892348.601598] [7ff5451e42d4:90253:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3815206Z ok (6.852s)
2022-12-01T11:08:36.3815226Z 
2022-12-01T11:08:36.3815487Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3815602Z Ran 1 test in 6.852s
2022-12-01T11:08:36.3815621Z 
2022-12-01T11:08:36.3815714Z OK
2022-12-01T11:08:36.3815733Z 
2022-12-01T11:08:36.3815855Z Generating XML reports...
2022-12-01T11:08:36.3816308Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105903.xml
2022-12-01T11:08:36.3816683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3816864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3817249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3817495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3817516Z 
2022-12-01T11:08:36.3817612Z Running tests...
2022-12-01T11:08:36.3817882Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3818197Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3818470Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3818693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90554
2022-12-01T11:08:36.3818918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90555
2022-12-01T11:08:36.3819292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3819477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3819844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3820037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3820405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3820581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3820956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3821144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3821393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3821636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3822105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3822489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3822723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3822948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3823204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp904sz4qq
2022-12-01T11:08:36.3823473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp904sz4qq/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3823789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ucu7z7f
2022-12-01T11:08:36.3824064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ucu7z7f/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3824310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3824527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3824764Z [1669892358.078822] [7ff5451e42d4:90555:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3825005Z [1669892358.078822] [7ff5451e42d4:90555:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3825243Z [1669892358.074537] [7ff5451e42d4:90554:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3825477Z [1669892358.074537] [7ff5451e42d4:90554:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3825891Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp7ucu7z7f'>
2022-12-01T11:08:36.3826113Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:08:36.3826521Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp904sz4qq'>
2022-12-01T11:08:36.3826684Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:08:36.3826769Z ok (6.379s)
2022-12-01T11:08:36.3826809Z 
2022-12-01T11:08:36.3827062Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3827176Z Ran 1 test in 6.380s
2022-12-01T11:08:36.3827195Z 
2022-12-01T11:08:36.3827289Z OK
2022-12-01T11:08:36.3827308Z 
2022-12-01T11:08:36.3827433Z Generating XML reports...
2022-12-01T11:08:36.3827883Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105912.xml
2022-12-01T11:08:36.3828259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3828448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3828836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3829010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3829029Z 
2022-12-01T11:08:36.3829140Z Running tests...
2022-12-01T11:08:36.3829651Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3829982Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3830246Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3830471Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90924
2022-12-01T11:08:36.3830692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90925
2022-12-01T11:08:36.3831164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3831324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3831709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3831907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3832275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3832452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3832826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3833017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3833269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3833524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3833911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3834311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3834544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3834804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzzjhm1ew
2022-12-01T11:08:36.3835079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzzjhm1ew/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3835310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3835571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzmz48lhi
2022-12-01T11:08:36.3835905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzmz48lhi/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3836160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3836379Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3836618Z [1669892366.447510] [7ff5451e42d4:90924:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3836862Z [1669892366.447510] [7ff5451e42d4:90924:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3837094Z [1669892366.451294] [7ff5451e42d4:90925:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3837331Z [1669892366.451294] [7ff5451e42d4:90925:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3837442Z ok (6.288s)
2022-12-01T11:08:36.3837464Z 
2022-12-01T11:08:36.3837741Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3837855Z Ran 1 test in 6.288s
2022-12-01T11:08:36.3837875Z 
2022-12-01T11:08:36.3837971Z OK
2022-12-01T11:08:36.3837991Z 
2022-12-01T11:08:36.3838098Z Generating XML reports...
2022-12-01T11:08:36.3838554Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105921.xml
2022-12-01T11:08:36.3838931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3839109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3839492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3839688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3839756Z 
2022-12-01T11:08:36.3839877Z Running tests...
2022-12-01T11:08:36.3840147Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3840442Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3840714Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3840940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91281
2022-12-01T11:08:36.3841161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91282
2022-12-01T11:08:36.3841538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3841720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3842104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3842303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3842673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3842832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3843209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3843401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3843651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3843895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3844296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3844747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3844991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3845216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3845457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytt1cx5x
2022-12-01T11:08:36.3845730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytt1cx5x/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3845987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jiotmcr
2022-12-01T11:08:36.3846260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jiotmcr/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3846496Z [1669892374.758268] [7ff5451e42d4:91281:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3846746Z [1669892374.758268] [7ff5451e42d4:91281:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3846977Z [1669892374.765125] [7ff5451e42d4:91282:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3847215Z [1669892374.765125] [7ff5451e42d4:91282:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3847457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3847676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3847782Z ok (6.566s)
2022-12-01T11:08:36.3847802Z 
2022-12-01T11:08:36.3848075Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3848190Z Ran 1 test in 6.566s
2022-12-01T11:08:36.3848257Z 
2022-12-01T11:08:36.3848355Z OK
2022-12-01T11:08:36.3848374Z 
2022-12-01T11:08:36.3848506Z Generating XML reports...
2022-12-01T11:08:36.3848960Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105929.xml
2022-12-01T11:08:36.3849335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3849494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3849878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3850074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3850094Z 
2022-12-01T11:08:36.3850205Z Running tests...
2022-12-01T11:08:36.3850472Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3850784Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3851238Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3851260Z 
2022-12-01T11:08:36.3851524Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3851642Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3851661Z 
2022-12-01T11:08:36.3851750Z OK (skipped=1)
2022-12-01T11:08:36.3851790Z 
2022-12-01T11:08:36.3851897Z Generating XML reports...
2022-12-01T11:08:36.3852346Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105938.xml
2022-12-01T11:08:36.3852721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3852900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3853283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3853542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3853563Z 
2022-12-01T11:08:36.3853677Z Running tests...
2022-12-01T11:08:36.3853948Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3854241Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3854669Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3854688Z 
2022-12-01T11:08:36.3854952Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3855064Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3855083Z 
2022-12-01T11:08:36.3855192Z OK (skipped=1)
2022-12-01T11:08:36.3855211Z 
2022-12-01T11:08:36.3855335Z Generating XML reports...
2022-12-01T11:08:36.3855787Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105940.xml
2022-12-01T11:08:36.3856160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3856343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3856705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3856899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3856918Z 
2022-12-01T11:08:36.3857027Z Running tests...
2022-12-01T11:08:36.3857297Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3857611Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3858079Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3858150Z 
2022-12-01T11:08:36.3858424Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3858538Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3858557Z 
2022-12-01T11:08:36.3858666Z OK (skipped=1)
2022-12-01T11:08:36.3858685Z 
2022-12-01T11:08:36.3858791Z Generating XML reports...
2022-12-01T11:08:36.3859236Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105942.xml
2022-12-01T11:08:36.3859610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3859787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3860171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3860363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3860386Z 
2022-12-01T11:08:36.3860500Z Running tests...
2022-12-01T11:08:36.3860767Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3861065Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3861510Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3861530Z 
2022-12-01T11:08:36.3861794Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3861907Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3861926Z 
2022-12-01T11:08:36.3862035Z OK (skipped=1)
2022-12-01T11:08:36.3862054Z 
2022-12-01T11:08:36.3862177Z Generating XML reports...
2022-12-01T11:08:36.3862617Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105944.xml
2022-12-01T11:08:36.3863044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3863229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3863596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3863789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3863809Z 
2022-12-01T11:08:36.3863918Z Running tests...
2022-12-01T11:08:36.3864181Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3864490Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3864883Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-12-01T11:08:36.3864902Z 
2022-12-01T11:08:36.3865168Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3865287Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3865309Z 
2022-12-01T11:08:36.3865418Z OK (skipped=1)
2022-12-01T11:08:36.3865437Z 
2022-12-01T11:08:36.3865543Z Generating XML reports...
2022-12-01T11:08:36.3865998Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105946.xml
2022-12-01T11:08:36.3866375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3866555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3866939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3867134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3867154Z 
2022-12-01T11:08:36.3867263Z Running tests...
2022-12-01T11:08:36.3867528Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3867921Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3868166Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3868388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91977
2022-12-01T11:08:36.3868607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91978
2022-12-01T11:08:36.3868981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3869156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3869778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3869981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3870373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3870528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3870946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3871136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3871385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3871636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3872042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3872446Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3872760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3873005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3873244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiet4k1fp
2022-12-01T11:08:36.3873517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiet4k1fp/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3873770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qb7cwut
2022-12-01T11:08:36.3874041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qb7cwut/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3874278Z [1669892394.040432] [7ff5451e42d4:91978:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3874517Z [1669892394.040432] [7ff5451e42d4:91978:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3875305Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:08:36.3875541Z [1669892394.035553] [7ff5451e42d4:91977:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3875780Z [1669892394.035553] [7ff5451e42d4:91977:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3876558Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:08:36.3876873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3877114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3877219Z ok (6.679s)
2022-12-01T11:08:36.3877239Z 
2022-12-01T11:08:36.3877497Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3877612Z Ran 1 test in 6.679s
2022-12-01T11:08:36.3877631Z 
2022-12-01T11:08:36.3877724Z OK
2022-12-01T11:08:36.3877747Z 
2022-12-01T11:08:36.3877876Z Generating XML reports...
2022-12-01T11:08:36.3878333Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105948.xml
2022-12-01T11:08:36.3878711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3878891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3879274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3879447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3879487Z 
2022-12-01T11:08:36.3879579Z Running tests...
2022-12-01T11:08:36.3879843Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3880154Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3880494Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3881261Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.815s)
2022-12-01T11:08:36.3881282Z 
2022-12-01T11:08:36.3881551Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3881665Z Ran 1 test in 1.815s
2022-12-01T11:08:36.3881684Z 
2022-12-01T11:08:36.3881793Z OK (skipped=1)
2022-12-01T11:08:36.3881811Z 
2022-12-01T11:08:36.3881938Z Generating XML reports...
2022-12-01T11:08:36.3882364Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105957.xml
2022-12-01T11:08:36.3882739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3882925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3883312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3883506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3883527Z 
2022-12-01T11:08:36.3883638Z Running tests...
2022-12-01T11:08:36.3883907Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3884224Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3884653Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3884674Z 
2022-12-01T11:08:36.3884916Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3885087Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3885107Z 
2022-12-01T11:08:36.3885215Z OK (skipped=1)
2022-12-01T11:08:36.3885237Z 
2022-12-01T11:08:36.3885363Z Generating XML reports...
2022-12-01T11:08:36.3885814Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110001.xml
2022-12-01T11:08:36.3886189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3886369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3886751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3886929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3886969Z 
2022-12-01T11:08:36.3887059Z Running tests...
2022-12-01T11:08:36.3887323Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3887639Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3888070Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.3888090Z 
2022-12-01T11:08:36.3888353Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3888466Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3888485Z 
2022-12-01T11:08:36.3888593Z OK (skipped=1)
2022-12-01T11:08:36.3888611Z 
2022-12-01T11:08:36.3888737Z Generating XML reports...
2022-12-01T11:08:36.3889170Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110004.xml
2022-12-01T11:08:36.3889545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3889722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3890156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3890358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3890377Z 
2022-12-01T11:08:36.3890488Z Running tests...
2022-12-01T11:08:36.3890758Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3891074Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3891342Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3891547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92682
2022-12-01T11:08:36.3891771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92683
2022-12-01T11:08:36.3892143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3892326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3892713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3892906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3893269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3893445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3893802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3893998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3894244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3894490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3894961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3895360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3895596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3895828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3896087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoir1cuut
2022-12-01T11:08:36.3896338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoir1cuut/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3896594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fedkvo3
2022-12-01T11:08:36.3896868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fedkvo3/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3897106Z [1669892411.421446] [7ff5451e42d4:92683:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3897346Z [1669892411.421446] [7ff5451e42d4:92683:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3897572Z [1669892411.414751] [7ff5451e42d4:92682:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3897811Z [1669892411.414751] [7ff5451e42d4:92682:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3897914Z ok (6.156s)
2022-12-01T11:08:36.3897935Z 
2022-12-01T11:08:36.3898208Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3898303Z Ran 1 test in 6.156s
2022-12-01T11:08:36.3898342Z 
2022-12-01T11:08:36.3898421Z OK
2022-12-01T11:08:36.3898440Z 
2022-12-01T11:08:36.3898567Z Generating XML reports...
2022-12-01T11:08:36.3899070Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110006.xml
2022-12-01T11:08:36.3899464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3899639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3900024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3900217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3900236Z 
2022-12-01T11:08:36.3900345Z Running tests...
2022-12-01T11:08:36.3900589Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3900910Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3901199Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3901953Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.739s)
2022-12-01T11:08:36.3901973Z 
2022-12-01T11:08:36.3902238Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3902352Z Ran 1 test in 1.739s
2022-12-01T11:08:36.3902371Z 
2022-12-01T11:08:36.3902480Z OK (skipped=1)
2022-12-01T11:08:36.3902499Z 
2022-12-01T11:08:36.3902624Z Generating XML reports...
2022-12-01T11:08:36.3903072Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110014.xml
2022-12-01T11:08:36.3903448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3903670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3904058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3904251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3904271Z 
2022-12-01T11:08:36.3904380Z Running tests...
2022-12-01T11:08:36.3904645Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3904958Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3905390Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T11:08:36.3905410Z 
2022-12-01T11:08:36.3905679Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3905779Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3905818Z 
2022-12-01T11:08:36.3905912Z OK (skipped=1)
2022-12-01T11:08:36.3905931Z 
2022-12-01T11:08:36.3906056Z Generating XML reports...
2022-12-01T11:08:36.3906506Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110018.xml
2022-12-01T11:08:36.3906879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3907055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3907438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3907633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3907653Z 
2022-12-01T11:08:36.3907762Z Running tests...
2022-12-01T11:08:36.3908006Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3908376Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3908664Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3909705Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.751s)
2022-12-01T11:08:36.3909727Z 
2022-12-01T11:08:36.3910013Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3910130Z Ran 1 test in 1.751s
2022-12-01T11:08:36.3910149Z 
2022-12-01T11:08:36.3910259Z OK (skipped=1)
2022-12-01T11:08:36.3910278Z 
2022-12-01T11:08:36.3910403Z Generating XML reports...
2022-12-01T11:08:36.3910856Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110020.xml
2022-12-01T11:08:36.3911240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3911400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3911786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3911980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3912000Z 
2022-12-01T11:08:36.3912111Z Running tests...
2022-12-01T11:08:36.3912380Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3912696Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3912973Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3913302Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93355
2022-12-01T11:08:36.3913504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93356
2022-12-01T11:08:36.3913886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3914067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3914451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3914644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3915011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3915185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3915560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3915761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3915993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3916245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3916652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3917050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3917281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3917512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3917774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppx2m4r4m
2022-12-01T11:08:36.3918114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppx2m4r4m/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3918381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk7z4sffe
2022-12-01T11:08:36.3918630Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk7z4sffe/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3918869Z [1669892429.604256] [7ff5451e42d4:93356:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3919109Z [1669892429.604256] [7ff5451e42d4:93356:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3919472Z STAGE:2022-12-01 11:00:30 93356:93356 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3919708Z [1669892429.602987] [7ff5451e42d4:93355:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3919955Z [1669892429.602987] [7ff5451e42d4:93355:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3920299Z STAGE:2022-12-01 11:00:30 93355:93355 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3920539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3920767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:08:36.3921094Z STAGE:2022-12-01 11:00:30 93355:93355 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3921431Z STAGE:2022-12-01 11:00:30 93356:93356 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3921763Z STAGE:2022-12-01 11:00:30 93355:93355 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.3922095Z STAGE:2022-12-01 11:00:31 93355:93355 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.3922260Z ok (7.270s)
2022-12-01T11:08:36.3922283Z 
2022-12-01T11:08:36.3922554Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3922672Z Ran 1 test in 7.270s
2022-12-01T11:08:36.3922691Z 
2022-12-01T11:08:36.3922786Z OK
2022-12-01T11:08:36.3922805Z 
2022-12-01T11:08:36.3922911Z Generating XML reports...
2022-12-01T11:08:36.3923368Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110024.xml
2022-12-01T11:08:36.3923746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3923925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3924308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3924502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3924525Z 
2022-12-01T11:08:36.3924637Z Running tests...
2022-12-01T11:08:36.3924905Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3925219Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3925470Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3925694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93724
2022-12-01T11:08:36.3925914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93725
2022-12-01T11:08:36.3926290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3926468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3926850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3927098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3927481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3927637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3928012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3928204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3928452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3928698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3929103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3929514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3929747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3929978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3930218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp12qy2li2
2022-12-01T11:08:36.3930489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp12qy2li2/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3930742Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvd_mg5_0
2022-12-01T11:08:36.3931009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvd_mg5_0/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3931244Z [1669892439.227237] [7ff5451e42d4:93725:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3931617Z [1669892439.227237] [7ff5451e42d4:93725:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3931845Z [1669892439.226114] [7ff5451e42d4:93724:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3932081Z [1669892439.226114] [7ff5451e42d4:93724:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3932186Z ok (5.996s)
2022-12-01T11:08:36.3932206Z 
2022-12-01T11:08:36.3932483Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3932580Z Ran 1 test in 5.996s
2022-12-01T11:08:36.3932599Z 
2022-12-01T11:08:36.3932693Z OK
2022-12-01T11:08:36.3932712Z 
2022-12-01T11:08:36.3932838Z Generating XML reports...
2022-12-01T11:08:36.3933292Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110034.xml
2022-12-01T11:08:36.3933677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3933861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3934246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3934441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3934461Z 
2022-12-01T11:08:36.3934552Z Running tests...
2022-12-01T11:08:36.3934821Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3935138Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3935418Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3935640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94072
2022-12-01T11:08:36.3935911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94073
2022-12-01T11:08:36.3936298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3936477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3936844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3937037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3937401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3937578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3937953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3938149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3938396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3938641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3939049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3939435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3939671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3939902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3940164Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj8nqzr66
2022-12-01T11:08:36.3940498Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj8nqzr66/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3940756Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjljo3zq
2022-12-01T11:08:36.3941025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjljo3zq/_remote_module_non_scriptable.py
2022-12-01T11:08:36.3941262Z [1669892447.546570] [7ff5451e42d4:94073:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3941502Z [1669892447.546570] [7ff5451e42d4:94073:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3941714Z [1669892447.540152] [7ff5451e42d4:94072:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.3941955Z [1669892447.540152] [7ff5451e42d4:94072:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.3942067Z ok (5.954s)
2022-12-01T11:08:36.3942087Z 
2022-12-01T11:08:36.3942368Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3942483Z Ran 1 test in 5.954s
2022-12-01T11:08:36.3942502Z 
2022-12-01T11:08:36.3942597Z OK
2022-12-01T11:08:36.3942617Z 
2022-12-01T11:08:36.3942741Z Generating XML reports...
2022-12-01T11:08:36.3943194Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110042.xml
2022-12-01T11:08:36.3943571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3943730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3944113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3944305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3944328Z 
2022-12-01T11:08:36.3944440Z Running tests...
2022-12-01T11:08:36.3944752Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3945078Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3945355Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3946103Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.735s)
2022-12-01T11:08:36.3946124Z 
2022-12-01T11:08:36.3946392Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3946487Z Ran 1 test in 1.735s
2022-12-01T11:08:36.3946527Z 
2022-12-01T11:08:36.3946621Z OK (skipped=1)
2022-12-01T11:08:36.3946640Z 
2022-12-01T11:08:36.3946766Z Generating XML reports...
2022-12-01T11:08:36.3947225Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110050.xml
2022-12-01T11:08:36.3947605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3947783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3948165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3948360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3948380Z 
2022-12-01T11:08:36.3948490Z Running tests...
2022-12-01T11:08:36.3948734Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3949047Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3949605Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3950385Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.678s)
2022-12-01T11:08:36.3950406Z 
2022-12-01T11:08:36.3950674Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3950790Z Ran 1 test in 1.678s
2022-12-01T11:08:36.3950810Z 
2022-12-01T11:08:36.3950922Z OK (skipped=1)
2022-12-01T11:08:36.3950940Z 
2022-12-01T11:08:36.3951065Z Generating XML reports...
2022-12-01T11:08:36.3951512Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110054.xml
2022-12-01T11:08:36.3951889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3952062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3952451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3952644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3952663Z 
2022-12-01T11:08:36.3952772Z Running tests...
2022-12-01T11:08:36.3953040Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3953352Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3953641Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3954471Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.745s)
2022-12-01T11:08:36.3954496Z 
2022-12-01T11:08:36.3954782Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3954877Z Ran 1 test in 1.746s
2022-12-01T11:08:36.3954916Z 
2022-12-01T11:08:36.3955006Z OK (skipped=1)
2022-12-01T11:08:36.3955024Z 
2022-12-01T11:08:36.3955150Z Generating XML reports...
2022-12-01T11:08:36.3955602Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110058.xml
2022-12-01T11:08:36.3955977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3956156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3956535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3956734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3956753Z 
2022-12-01T11:08:36.3956864Z Running tests...
2022-12-01T11:08:36.3957110Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3957423Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3957878Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.003s)
2022-12-01T11:08:36.3957898Z 
2022-12-01T11:08:36.3958165Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3958280Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3958299Z 
2022-12-01T11:08:36.3958408Z OK (skipped=1)
2022-12-01T11:08:36.3958427Z 
2022-12-01T11:08:36.3958551Z Generating XML reports...
2022-12-01T11:08:36.3959004Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110102.xml
2022-12-01T11:08:36.3959459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3959620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3960005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3960199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3960218Z 
2022-12-01T11:08:36.3960331Z Running tests...
2022-12-01T11:08:36.3960599Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3960912Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3961176Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3961406Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94874
2022-12-01T11:08:36.3961610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94875
2022-12-01T11:08:36.3961988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3962169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3962556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3962748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3963117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3963290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3963669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3963918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3964155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3964402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3964809Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3965211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3965449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3965681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3965923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3966172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3966573Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3966953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3967061Z ok (3.836s)
2022-12-01T11:08:36.3967082Z 
2022-12-01T11:08:36.3967350Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3967466Z Ran 1 test in 3.836s
2022-12-01T11:08:36.3967485Z 
2022-12-01T11:08:36.3967579Z OK
2022-12-01T11:08:36.3967598Z 
2022-12-01T11:08:36.3967723Z Generating XML reports...
2022-12-01T11:08:36.3968177Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110104.xml
2022-12-01T11:08:36.3968621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3968798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3969161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3969356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3969375Z 
2022-12-01T11:08:36.3969485Z Running tests...
2022-12-01T11:08:36.3969751Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3970067Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3970325Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3970547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95171
2022-12-01T11:08:36.3970820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95172
2022-12-01T11:08:36.3971183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3971361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3971745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3971938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3972312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3972488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3972864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3973061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3973357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3973593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3973996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3974395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3974628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3974872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.3975097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3975342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.3975747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3976146Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.3976232Z ok (4.273s)
2022-12-01T11:08:36.3976252Z 
2022-12-01T11:08:36.3976519Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3976635Z Ran 1 test in 4.274s
2022-12-01T11:08:36.3976654Z 
2022-12-01T11:08:36.3976746Z OK
2022-12-01T11:08:36.3976765Z 
2022-12-01T11:08:36.3976891Z Generating XML reports...
2022-12-01T11:08:36.3977340Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110110.xml
2022-12-01T11:08:36.3977716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3977957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3978328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3978522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3978542Z 
2022-12-01T11:08:36.3978653Z Running tests...
2022-12-01T11:08:36.3978916Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3979230Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3979509Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3980261Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.685s)
2022-12-01T11:08:36.3980285Z 
2022-12-01T11:08:36.3980553Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3980667Z Ran 1 test in 1.685s
2022-12-01T11:08:36.3980686Z 
2022-12-01T11:08:36.3980794Z OK (skipped=1)
2022-12-01T11:08:36.3980813Z 
2022-12-01T11:08:36.3980919Z Generating XML reports...
2022-12-01T11:08:36.3981370Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110117.xml
2022-12-01T11:08:36.3981748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3981924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3982308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3982549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3982571Z 
2022-12-01T11:08:36.3982685Z Running tests...
2022-12-01T11:08:36.3982953Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3983245Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3983669Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.3983689Z 
2022-12-01T11:08:36.3983959Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3984073Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3984092Z 
2022-12-01T11:08:36.3984199Z OK (skipped=1)
2022-12-01T11:08:36.3984218Z 
2022-12-01T11:08:36.3984343Z Generating XML reports...
2022-12-01T11:08:36.3984795Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110121.xml
2022-12-01T11:08:36.3985182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3985359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3985718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3985910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3985930Z 
2022-12-01T11:08:36.3986038Z Running tests...
2022-12-01T11:08:36.3986302Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3986615Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3986886Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.3987107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95705
2022-12-01T11:08:36.3987388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95706
2022-12-01T11:08:36.3987763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3987924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3988305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3988497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3988870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3989044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3989663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3989871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3990123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.3990351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.3990763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3991161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.3991394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.3991624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.3991727Z ok (4.062s)
2022-12-01T11:08:36.3991747Z 
2022-12-01T11:08:36.3992023Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3992218Z Ran 1 test in 4.062s
2022-12-01T11:08:36.3992240Z 
2022-12-01T11:08:36.3992343Z OK
2022-12-01T11:08:36.3992362Z 
2022-12-01T11:08:36.3992471Z Generating XML reports...
2022-12-01T11:08:36.3992924Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110123.xml
2022-12-01T11:08:36.3993298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3993474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3993856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3994049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3994069Z 
2022-12-01T11:08:36.3994180Z Running tests...
2022-12-01T11:08:36.3994454Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3994779Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3995022Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.3995042Z 
2022-12-01T11:08:36.3995312Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3995425Z Ran 1 test in 0.002s
2022-12-01T11:08:36.3995444Z 
2022-12-01T11:08:36.3995555Z OK (skipped=1)
2022-12-01T11:08:36.3995574Z 
2022-12-01T11:08:36.3995700Z Generating XML reports...
2022-12-01T11:08:36.3996147Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110129.xml
2022-12-01T11:08:36.3996519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.3996698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.3997164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.3997361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.3997381Z 
2022-12-01T11:08:36.3997490Z Running tests...
2022-12-01T11:08:36.3997756Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3998068Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.3998342Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.3998361Z 
2022-12-01T11:08:36.3998626Z ----------------------------------------------------------------------
2022-12-01T11:08:36.3998740Z Ran 1 test in 0.003s
2022-12-01T11:08:36.3998760Z 
2022-12-01T11:08:36.3998870Z OK (skipped=1)
2022-12-01T11:08:36.3998889Z 
2022-12-01T11:08:36.3998998Z Generating XML reports...
2022-12-01T11:08:36.3999448Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110132.xml
2022-12-01T11:08:36.3999824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4000001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4000382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4000575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4000595Z 
2022-12-01T11:08:36.4000707Z Running tests...
2022-12-01T11:08:36.4000974Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4001290Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4001530Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T11:08:36.4001600Z 
2022-12-01T11:08:36.4001877Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4001993Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4002012Z 
2022-12-01T11:08:36.4002120Z OK (skipped=1)
2022-12-01T11:08:36.4002139Z 
2022-12-01T11:08:36.4002263Z Generating XML reports...
2022-12-01T11:08:36.4002716Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110134.xml
2022-12-01T11:08:36.4003087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4003266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4003650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4003825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4003849Z 
2022-12-01T11:08:36.4003962Z Running tests...
2022-12-01T11:08:36.4004228Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4004543Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4004818Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4004838Z 
2022-12-01T11:08:36.4005103Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4005217Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4005236Z 
2022-12-01T11:08:36.4005346Z OK (skipped=1)
2022-12-01T11:08:36.4005365Z 
2022-12-01T11:08:36.4005470Z Generating XML reports...
2022-12-01T11:08:36.4005916Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110136.xml
2022-12-01T11:08:36.4006287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4006524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4006914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4007106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4007125Z 
2022-12-01T11:08:36.4007238Z Running tests...
2022-12-01T11:08:36.4007502Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4007819Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4008071Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4008092Z 
2022-12-01T11:08:36.4008356Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4008477Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4008496Z 
2022-12-01T11:08:36.4008607Z OK (skipped=1)
2022-12-01T11:08:36.4008626Z 
2022-12-01T11:08:36.4008750Z Generating XML reports...
2022-12-01T11:08:36.4009197Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110138.xml
2022-12-01T11:08:36.4009571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4009751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4010133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4010307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4010326Z 
2022-12-01T11:08:36.4010436Z Running tests...
2022-12-01T11:08:36.4010701Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4011064Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4011469Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.4011489Z 
2022-12-01T11:08:36.4011747Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4011862Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4011881Z 
2022-12-01T11:08:36.4011990Z OK (skipped=1)
2022-12-01T11:08:36.4012010Z 
2022-12-01T11:08:36.4012134Z Generating XML reports...
2022-12-01T11:08:36.4012569Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110140.xml
2022-12-01T11:08:36.4012942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4013120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4013511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4013704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4013723Z 
2022-12-01T11:08:36.4013832Z Running tests...
2022-12-01T11:08:36.4014099Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4014414Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4014807Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.4014849Z 
2022-12-01T11:08:36.4015094Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4015207Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4015226Z 
2022-12-01T11:08:36.4015336Z OK (skipped=1)
2022-12-01T11:08:36.4015355Z 
2022-12-01T11:08:36.4015480Z Generating XML reports...
2022-12-01T11:08:36.4015998Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110142.xml
2022-12-01T11:08:36.4016374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4016551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4016935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4017109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4017149Z 
2022-12-01T11:08:36.4017239Z Running tests...
2022-12-01T11:08:36.4017508Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4017818Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4018065Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4018295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96541
2022-12-01T11:08:36.4018515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96542
2022-12-01T11:08:36.4018889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4019046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4019428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4019620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4019989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4020165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4020597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4020797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4021046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4021291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4021679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4022077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4022310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4022556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4022791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4023028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4023429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4023826Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4023931Z ok (9.636s)
2022-12-01T11:08:36.4023952Z 
2022-12-01T11:08:36.4024200Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4024315Z Ran 1 test in 9.636s
2022-12-01T11:08:36.4024334Z 
2022-12-01T11:08:36.4024428Z OK
2022-12-01T11:08:36.4024447Z 
2022-12-01T11:08:36.4024571Z Generating XML reports...
2022-12-01T11:08:36.4025023Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110144.xml
2022-12-01T11:08:36.4025462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4025642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4026026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4026219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4026239Z 
2022-12-01T11:08:36.4026331Z Running tests...
2022-12-01T11:08:36.4026598Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4026917Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4027197Z   test_get_future (__main__.TestDistBackendWithSpawn) ... skip: get_future is only supported on mpi, nccl and gloo (0.002s)
2022-12-01T11:08:36.4027221Z 
2022-12-01T11:08:36.4027489Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4027606Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4027625Z 
2022-12-01T11:08:36.4027735Z OK (skipped=1)
2022-12-01T11:08:36.4027754Z 
2022-12-01T11:08:36.4027879Z Generating XML reports...
2022-12-01T11:08:36.4028306Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110156.xml
2022-12-01T11:08:36.4028680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4028858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4029240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4029626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4029647Z 
2022-12-01T11:08:36.4029771Z Running tests...
2022-12-01T11:08:36.4030045Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4030436Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4030694Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4030896Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96909
2022-12-01T11:08:36.4031117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96910
2022-12-01T11:08:36.4031498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4031678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4032059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4032254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4032632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4032812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4033176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4033368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4033617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4033862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4034273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4034673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4034986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4035217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4035319Z ok (4.268s)
2022-12-01T11:08:36.4035339Z 
2022-12-01T11:08:36.4035591Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4035704Z Ran 1 test in 4.268s
2022-12-01T11:08:36.4035723Z 
2022-12-01T11:08:36.4035818Z OK
2022-12-01T11:08:36.4035838Z 
2022-12-01T11:08:36.4035963Z Generating XML reports...
2022-12-01T11:08:36.4036415Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110158.xml
2022-12-01T11:08:36.4036787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4036964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4037353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4037548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4037568Z 
2022-12-01T11:08:36.4037659Z Running tests...
2022-12-01T11:08:36.4037930Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4038247Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4038516Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4038738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97230
2022-12-01T11:08:36.4038956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97231
2022-12-01T11:08:36.4039330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4039560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4039932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4040125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4040491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4040668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4041044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4041234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4041486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4041741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4042148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4042534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4042766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4043009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4043235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4043472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4043875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4044336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4044441Z ok (4.198s)
2022-12-01T11:08:36.4044460Z 
2022-12-01T11:08:36.4044733Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4044828Z Ran 1 test in 4.198s
2022-12-01T11:08:36.4044847Z 
2022-12-01T11:08:36.4044941Z OK
2022-12-01T11:08:36.4044960Z 
2022-12-01T11:08:36.4045084Z Generating XML reports...
2022-12-01T11:08:36.4045538Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110204.xml
2022-12-01T11:08:36.4045911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4046090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4046473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4046675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4046694Z 
2022-12-01T11:08:36.4046786Z Running tests...
2022-12-01T11:08:36.4047052Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4047367Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4047634Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4047861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97447
2022-12-01T11:08:36.4048079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97448
2022-12-01T11:08:36.4048456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4048638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4049067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4049249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4049623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4049800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4050179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4050370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4050615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4050862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4051276Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4051660Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4051896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4052140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4052364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4052602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4053009Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4053473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4053580Z ok (4.188s)
2022-12-01T11:08:36.4053599Z 
2022-12-01T11:08:36.4053867Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4053961Z Ran 1 test in 4.188s
2022-12-01T11:08:36.4054002Z 
2022-12-01T11:08:36.4054076Z OK
2022-12-01T11:08:36.4054095Z 
2022-12-01T11:08:36.4054219Z Generating XML reports...
2022-12-01T11:08:36.4054668Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110211.xml
2022-12-01T11:08:36.4055041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4055221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4055604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4055799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4055821Z 
2022-12-01T11:08:36.4055930Z Running tests...
2022-12-01T11:08:36.4056176Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4056492Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4056899Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.003s)
2022-12-01T11:08:36.4056920Z 
2022-12-01T11:08:36.4057182Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4057295Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4057316Z 
2022-12-01T11:08:36.4057425Z OK (skipped=1)
2022-12-01T11:08:36.4057444Z 
2022-12-01T11:08:36.4057567Z Generating XML reports...
2022-12-01T11:08:36.4058015Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110217.xml
2022-12-01T11:08:36.4058443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4058608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4058993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4059185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4059205Z 
2022-12-01T11:08:36.4059315Z Running tests...
2022-12-01T11:08:36.4059583Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4059898Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4060140Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4060362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97841
2022-12-01T11:08:36.4060568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97842
2022-12-01T11:08:36.4060944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4061121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4061504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4061698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4062064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4062240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4062614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4062880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4063115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4063360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4063766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4064165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4064399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4064628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4064864Z [1669892545.010721] [7ff5451e42d4:97842:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4065113Z [1669892545.010721] [7ff5451e42d4:97842:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4065348Z [1669892545.011112] [7ff5451e42d4:97841:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4065563Z [1669892545.011112] [7ff5451e42d4:97841:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4065667Z ok (6.124s)
2022-12-01T11:08:36.4065686Z 
2022-12-01T11:08:36.4065958Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4066074Z Ran 1 test in 6.125s
2022-12-01T11:08:36.4066094Z 
2022-12-01T11:08:36.4066187Z OK
2022-12-01T11:08:36.4066205Z 
2022-12-01T11:08:36.4066330Z Generating XML reports...
2022-12-01T11:08:36.4066784Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110219.xml
2022-12-01T11:08:36.4067211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4067378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4067767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4067962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4067982Z 
2022-12-01T11:08:36.4068092Z Running tests...
2022-12-01T11:08:36.4068359Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4068676Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4068917Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4069138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98183
2022-12-01T11:08:36.4069544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98184
2022-12-01T11:08:36.4069931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4070108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4070496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4070689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4071098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4071273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4071652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4071849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4072169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4072416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4072822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4073221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4073455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4073685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4073922Z [1669892553.427945] [7ff5451e42d4:98184:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4074170Z [1669892553.427945] [7ff5451e42d4:98184:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4074402Z [1669892553.413588] [7ff5451e42d4:98183:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4074632Z [1669892553.413588] [7ff5451e42d4:98183:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4074717Z ok (5.965s)
2022-12-01T11:08:36.4074737Z 
2022-12-01T11:08:36.4075008Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4075123Z Ran 1 test in 5.965s
2022-12-01T11:08:36.4075142Z 
2022-12-01T11:08:36.4075238Z OK
2022-12-01T11:08:36.4075256Z 
2022-12-01T11:08:36.4075382Z Generating XML reports...
2022-12-01T11:08:36.4075833Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110228.xml
2022-12-01T11:08:36.4076315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4076505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4076876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4077071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4077094Z 
2022-12-01T11:08:36.4077204Z Running tests...
2022-12-01T11:08:36.4077471Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4077784Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4078051Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4078274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98506
2022-12-01T11:08:36.4078498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98507
2022-12-01T11:08:36.4078879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4079038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4079419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4079611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4079987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4080167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4080545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4080737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4081049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4081274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4081682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4082079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4082311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4082651Z STAGE:2022-12-01 11:02:40 98506:98506 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4082882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4083217Z STAGE:2022-12-01 11:02:40 98507:98507 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4083463Z [1669892561.940387] [7ff5451e42d4:98506:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4083706Z [1669892561.940387] [7ff5451e42d4:98506:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4084050Z STAGE:2022-12-01 11:02:42 98506:98506 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4084266Z [1669892561.941011] [7ff5451e42d4:98507:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4084503Z [1669892561.941011] [7ff5451e42d4:98507:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4084846Z STAGE:2022-12-01 11:02:42 98507:98507 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4084950Z ok (7.147s)
2022-12-01T11:08:36.4084974Z 
2022-12-01T11:08:36.4085294Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4085417Z Ran 1 test in 7.147s
2022-12-01T11:08:36.4085437Z 
2022-12-01T11:08:36.4085530Z OK
2022-12-01T11:08:36.4085549Z 
2022-12-01T11:08:36.4085674Z Generating XML reports...
2022-12-01T11:08:36.4086114Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110236.xml
2022-12-01T11:08:36.4086490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4086670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4087054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4087247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4087266Z 
2022-12-01T11:08:36.4087379Z Running tests...
2022-12-01T11:08:36.4087645Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4087960Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4088227Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4088430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98893
2022-12-01T11:08:36.4088648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98894
2022-12-01T11:08:36.4089021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4089199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4089584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4089838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4090212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4090386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4090741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4090932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4091180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4091425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4091829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4092231Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4092469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4092810Z STAGE:2022-12-01 11:02:49 98893:98893 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4093041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4093354Z STAGE:2022-12-01 11:02:49 98894:98894 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4093592Z [1669892571.430674] [7ff5451e42d4:98893:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4093831Z [1669892571.430674] [7ff5451e42d4:98893:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4094169Z STAGE:2022-12-01 11:02:51 98893:98893 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4094456Z [1669892571.406005] [7ff5451e42d4:98894:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4094704Z [1669892571.406005] [7ff5451e42d4:98894:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4095050Z STAGE:2022-12-01 11:02:51 98894:98894 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4095155Z ok (7.253s)
2022-12-01T11:08:36.4095175Z 
2022-12-01T11:08:36.4095440Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4095536Z Ran 1 test in 7.253s
2022-12-01T11:08:36.4095575Z 
2022-12-01T11:08:36.4095649Z OK
2022-12-01T11:08:36.4095667Z 
2022-12-01T11:08:36.4095793Z Generating XML reports...
2022-12-01T11:08:36.4096244Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110245.xml
2022-12-01T11:08:36.4096628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4096805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4097186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4097381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4097400Z 
2022-12-01T11:08:36.4097511Z Running tests...
2022-12-01T11:08:36.4097759Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4098072Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4098505Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.4098525Z 
2022-12-01T11:08:36.4098785Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4098964Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4098986Z 
2022-12-01T11:08:36.4099097Z OK (skipped=1)
2022-12-01T11:08:36.4099116Z 
2022-12-01T11:08:36.4099246Z Generating XML reports...
2022-12-01T11:08:36.4099699Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110255.xml
2022-12-01T11:08:36.4100069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4100229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4100610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4100802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4100821Z 
2022-12-01T11:08:36.4100929Z Running tests...
2022-12-01T11:08:36.4101193Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4101515Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4101975Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo', 'nccl'} (0.002s)
2022-12-01T11:08:36.4101996Z 
2022-12-01T11:08:36.4102260Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4102372Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4102391Z 
2022-12-01T11:08:36.4102480Z OK (skipped=1)
2022-12-01T11:08:36.4102499Z 
2022-12-01T11:08:36.4102622Z Generating XML reports...
2022-12-01T11:08:36.4103075Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110257.xml
2022-12-01T11:08:36.4103445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4103627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4104066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4104270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4104290Z 
2022-12-01T11:08:36.4104400Z Running tests...
2022-12-01T11:08:36.4104649Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4104965Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4105379Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.4105399Z 
2022-12-01T11:08:36.4105665Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4105777Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4105795Z 
2022-12-01T11:08:36.4105904Z OK (skipped=1)
2022-12-01T11:08:36.4105928Z 
2022-12-01T11:08:36.4106054Z Generating XML reports...
2022-12-01T11:08:36.4106510Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110259.xml
2022-12-01T11:08:36.4106880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4107038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4107421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4107613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4107633Z 
2022-12-01T11:08:36.4107747Z Running tests...
2022-12-01T11:08:36.4108019Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4108334Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4108799Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.4108819Z 
2022-12-01T11:08:36.4109083Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4109197Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4109216Z 
2022-12-01T11:08:36.4109305Z OK (skipped=1)
2022-12-01T11:08:36.4109323Z 
2022-12-01T11:08:36.4109754Z Generating XML reports...
2022-12-01T11:08:36.4110216Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110301.xml
2022-12-01T11:08:36.4110593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4110771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4111154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4111357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4111378Z 
2022-12-01T11:08:36.4111490Z Running tests...
2022-12-01T11:08:36.4111759Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4112055Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4112482Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.4112502Z 
2022-12-01T11:08:36.4112764Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4112879Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4112899Z 
2022-12-01T11:08:36.4113008Z OK (skipped=1)
2022-12-01T11:08:36.4113027Z 
2022-12-01T11:08:36.4113152Z Generating XML reports...
2022-12-01T11:08:36.4113601Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110303.xml
2022-12-01T11:08:36.4114059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4114252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4114622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4114815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4114835Z 
2022-12-01T11:08:36.4114945Z Running tests...
2022-12-01T11:08:36.4115214Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4115528Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4115941Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.4115966Z 
2022-12-01T11:08:36.4116232Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4116350Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4116370Z 
2022-12-01T11:08:36.4116458Z OK (skipped=1)
2022-12-01T11:08:36.4116496Z 
2022-12-01T11:08:36.4116603Z Generating XML reports...
2022-12-01T11:08:36.4117051Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110305.xml
2022-12-01T11:08:36.4117427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4117602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4117982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4118177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4118196Z 
2022-12-01T11:08:36.4118398Z Running tests...
2022-12-01T11:08:36.4118671Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4118962Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4119373Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.4119393Z 
2022-12-01T11:08:36.4119656Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4119770Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4119789Z 
2022-12-01T11:08:36.4119898Z OK (skipped=1)
2022-12-01T11:08:36.4119916Z 
2022-12-01T11:08:36.4120041Z Generating XML reports...
2022-12-01T11:08:36.4120493Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110307.xml
2022-12-01T11:08:36.4120868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4121056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4121418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4121611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4121631Z 
2022-12-01T11:08:36.4121738Z Running tests...
2022-12-01T11:08:36.4122003Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4122317Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4122721Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T11:08:36.4122742Z 
2022-12-01T11:08:36.4123009Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4123127Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4123150Z 
2022-12-01T11:08:36.4123262Z OK (skipped=1)
2022-12-01T11:08:36.4123281Z 
2022-12-01T11:08:36.4123435Z Generating XML reports...
2022-12-01T11:08:36.4123961Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110309.xml
2022-12-01T11:08:36.4124337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4124514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4124892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4125084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4125104Z 
2022-12-01T11:08:36.4125215Z Running tests...
2022-12-01T11:08:36.4125479Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4125770Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4126187Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T11:08:36.4126207Z 
2022-12-01T11:08:36.4126474Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4126589Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4126609Z 
2022-12-01T11:08:36.4126718Z OK (skipped=1)
2022-12-01T11:08:36.4126736Z 
2022-12-01T11:08:36.4126860Z Generating XML reports...
2022-12-01T11:08:36.4127310Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110312.xml
2022-12-01T11:08:36.4127684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4127863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4128226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4128501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4128520Z 
2022-12-01T11:08:36.4128635Z Running tests...
2022-12-01T11:08:36.4128906Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4129215Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4129616Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T11:08:36.4129636Z 
2022-12-01T11:08:36.4129896Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4130009Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4130028Z 
2022-12-01T11:08:36.4130138Z OK (skipped=1)
2022-12-01T11:08:36.4130157Z 
2022-12-01T11:08:36.4130261Z Generating XML reports...
2022-12-01T11:08:36.4130708Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110314.xml
2022-12-01T11:08:36.4131090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4131271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4131651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4131844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4131863Z 
2022-12-01T11:08:36.4131972Z Running tests...
2022-12-01T11:08:36.4132239Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4132553Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4132934Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-12-01T11:08:36.4132957Z 
2022-12-01T11:08:36.4133270Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4133391Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4133410Z 
2022-12-01T11:08:36.4133520Z OK (skipped=1)
2022-12-01T11:08:36.4133539Z 
2022-12-01T11:08:36.4133664Z Generating XML reports...
2022-12-01T11:08:36.4134118Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110316.xml
2022-12-01T11:08:36.4134490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4134669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4135055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4135232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4135257Z 
2022-12-01T11:08:36.4135369Z Running tests...
2022-12-01T11:08:36.4135637Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4135950Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4136249Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-12-01T11:08:36.4136268Z 
2022-12-01T11:08:36.4136532Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4136645Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4136664Z 
2022-12-01T11:08:36.4136772Z OK (skipped=1)
2022-12-01T11:08:36.4136791Z 
2022-12-01T11:08:36.4136915Z Generating XML reports...
2022-12-01T11:08:36.4137343Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110318.xml
2022-12-01T11:08:36.4137720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4137957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4138343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4138535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4138554Z 
2022-12-01T11:08:36.4138664Z Running tests...
2022-12-01T11:08:36.4138936Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4139248Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4139483Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:08:36.4139525Z 
2022-12-01T11:08:36.4139772Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4139886Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4139910Z 
2022-12-01T11:08:36.4140017Z OK (skipped=1)
2022-12-01T11:08:36.4140035Z 
2022-12-01T11:08:36.4140162Z Generating XML reports...
2022-12-01T11:08:36.4140611Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110320.xml
2022-12-01T11:08:36.4140985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4141163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4141547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4141722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4141741Z 
2022-12-01T11:08:36.4141852Z Running tests...
2022-12-01T11:08:36.4142117Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4142431Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4142755Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:08:36.4142777Z 
2022-12-01T11:08:36.4143051Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4143165Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4143185Z 
2022-12-01T11:08:36.4143294Z OK (skipped=1)
2022-12-01T11:08:36.4143313Z 
2022-12-01T11:08:36.4143438Z Generating XML reports...
2022-12-01T11:08:36.4143866Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110322.xml
2022-12-01T11:08:36.4144240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4144417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4144800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4145001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4145020Z 
2022-12-01T11:08:36.4145131Z Running tests...
2022-12-01T11:08:36.4145401Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4145717Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4146010Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:08:36.4146050Z 
2022-12-01T11:08:36.4146298Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4146414Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4146433Z 
2022-12-01T11:08:36.4146541Z OK (skipped=1)
2022-12-01T11:08:36.4146560Z 
2022-12-01T11:08:36.4146684Z Generating XML reports...
2022-12-01T11:08:36.4147135Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110324.xml
2022-12-01T11:08:36.4147572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4147749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4148133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4148306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4148346Z 
2022-12-01T11:08:36.4148436Z Running tests...
2022-12-01T11:08:36.4148703Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4149017Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4149319Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4149807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100687
2022-12-01T11:08:36.4150034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100688
2022-12-01T11:08:36.4150421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4150599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4150963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4151158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4151528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4151702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4152078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4152346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4152615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4152863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4153251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4153652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4153883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4154113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4154223Z ok (4.029s)
2022-12-01T11:08:36.4154243Z 
2022-12-01T11:08:36.4154518Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4154633Z Ran 1 test in 4.030s
2022-12-01T11:08:36.4154653Z 
2022-12-01T11:08:36.4154747Z OK
2022-12-01T11:08:36.4154766Z 
2022-12-01T11:08:36.4154890Z Generating XML reports...
2022-12-01T11:08:36.4155322Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110326.xml
2022-12-01T11:08:36.4155694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4155872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4156257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4156453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4156538Z 
2022-12-01T11:08:36.4156659Z Running tests...
2022-12-01T11:08:36.4156934Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4157248Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4157546Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4157750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100872
2022-12-01T11:08:36.4157972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100874
2022-12-01T11:08:36.4158346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4158525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4158910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4159110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4159477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4159655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4160012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4160205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4160450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4160696Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4161100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4161557Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4161798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4162028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4162131Z ok (3.995s)
2022-12-01T11:08:36.4162150Z 
2022-12-01T11:08:36.4162402Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4162514Z Ran 1 test in 3.995s
2022-12-01T11:08:36.4162533Z 
2022-12-01T11:08:36.4162626Z OK
2022-12-01T11:08:36.4162646Z 
2022-12-01T11:08:36.4162768Z Generating XML reports...
2022-12-01T11:08:36.4163218Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110332.xml
2022-12-01T11:08:36.4163593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4163777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4164160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4164334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4164373Z 
2022-12-01T11:08:36.4164465Z Running tests...
2022-12-01T11:08:36.4164733Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4165045Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4165324Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:08:36.4165343Z 
2022-12-01T11:08:36.4165607Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4165721Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4165791Z 
2022-12-01T11:08:36.4165907Z OK (skipped=1)
2022-12-01T11:08:36.4165926Z 
2022-12-01T11:08:36.4166057Z Generating XML reports...
2022-12-01T11:08:36.4166489Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110338.xml
2022-12-01T11:08:36.4166862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4167040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4167421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4167612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4167636Z 
2022-12-01T11:08:36.4167744Z Running tests...
2022-12-01T11:08:36.4168010Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4168322Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4168631Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:08:36.4168651Z 
2022-12-01T11:08:36.4168895Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4169009Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4169028Z 
2022-12-01T11:08:36.4169135Z OK (skipped=1)
2022-12-01T11:08:36.4169154Z 
2022-12-01T11:08:36.4169278Z Generating XML reports...
2022-12-01T11:08:36.4169722Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110341.xml
2022-12-01T11:08:36.4170096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4170274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4170658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4170953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4170975Z 
2022-12-01T11:08:36.4171071Z Running tests...
2022-12-01T11:08:36.4171344Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4171660Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4171938Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4172694Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.671s)
2022-12-01T11:08:36.4172715Z 
2022-12-01T11:08:36.4172986Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4173105Z Ran 1 test in 1.671s
2022-12-01T11:08:36.4173125Z 
2022-12-01T11:08:36.4173235Z OK (skipped=1)
2022-12-01T11:08:36.4173253Z 
2022-12-01T11:08:36.4173379Z Generating XML reports...
2022-12-01T11:08:36.4173808Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110343.xml
2022-12-01T11:08:36.4174185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4174362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4174746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4174939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4174958Z 
2022-12-01T11:08:36.4175067Z Running tests...
2022-12-01T11:08:36.4175395Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4175712Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4175993Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4176197Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101512
2022-12-01T11:08:36.4176418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101513
2022-12-01T11:08:36.4176798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4176976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4177359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4177553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4177927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4178102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4178482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4178654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4178902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4179149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4179553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4179955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4180240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4180481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4180741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdq289yi1
2022-12-01T11:08:36.4180994Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdq289yi1/_remote_module_non_scriptable.py
2022-12-01T11:08:36.4181249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevife3re
2022-12-01T11:08:36.4181517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevife3re/_remote_module_non_scriptable.py
2022-12-01T11:08:36.4181753Z [1669892632.195761] [7ff5451e42d4:101512:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4181995Z [1669892632.195761] [7ff5451e42d4:101512:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4182227Z [1669892632.200825] [7ff5451e42d4:101513:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4182468Z [1669892632.200825] [7ff5451e42d4:101513:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4182578Z ok (6.518s)
2022-12-01T11:08:36.4182599Z 
2022-12-01T11:08:36.4182873Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4182968Z Ran 1 test in 6.518s
2022-12-01T11:08:36.4183007Z 
2022-12-01T11:08:36.4183082Z OK
2022-12-01T11:08:36.4183100Z 
2022-12-01T11:08:36.4183224Z Generating XML reports...
2022-12-01T11:08:36.4183677Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110347.xml
2022-12-01T11:08:36.4184052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4184293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4184682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4184876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4184896Z 
2022-12-01T11:08:36.4185004Z Running tests...
2022-12-01T11:08:36.4185250Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4185567Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4185838Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4186062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101863
2022-12-01T11:08:36.4186282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101864
2022-12-01T11:08:36.4186667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4186848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4187236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4187430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4187781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4187958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4188336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4188530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4188828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4189083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4189736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4190153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4190364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4190593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4190829Z [1669892641.951267] [7ff5451e42d4:101863:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4191070Z [1669892641.951267] [7ff5451e42d4:101863:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4191312Z [1669892641.951263] [7ff5451e42d4:101864:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4191548Z [1669892641.951263] [7ff5451e42d4:101864:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4191650Z ok (7.139s)
2022-12-01T11:08:36.4191669Z 
2022-12-01T11:08:36.4191942Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4192059Z Ran 1 test in 7.140s
2022-12-01T11:08:36.4192078Z 
2022-12-01T11:08:36.4192152Z OK
2022-12-01T11:08:36.4192191Z 
2022-12-01T11:08:36.4192296Z Generating XML reports...
2022-12-01T11:08:36.4192745Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110355.xml
2022-12-01T11:08:36.4193119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4193403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4193792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4193985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4194004Z 
2022-12-01T11:08:36.4194114Z Running tests...
2022-12-01T11:08:36.4194382Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4194678Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4194970Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4195194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102179
2022-12-01T11:08:36.4195413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102180
2022-12-01T11:08:36.4195798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4195976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4196357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4196548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4196894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4197073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4197453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4197643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4197954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4198212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4198619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4199018Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4199253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4199464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4199702Z [1669892650.860808] [7ff5451e42d4:102179:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4199942Z [1669892650.860808] [7ff5451e42d4:102179:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4200182Z [1669892650.861779] [7ff5451e42d4:102180:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4200421Z [1669892650.861779] [7ff5451e42d4:102180:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4200527Z ok (6.997s)
2022-12-01T11:08:36.4200547Z 
2022-12-01T11:08:36.4200817Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4200933Z Ran 1 test in 6.997s
2022-12-01T11:08:36.4200953Z 
2022-12-01T11:08:36.4201046Z OK
2022-12-01T11:08:36.4201065Z 
2022-12-01T11:08:36.4201169Z Generating XML reports...
2022-12-01T11:08:36.4201620Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110404.xml
2022-12-01T11:08:36.4201995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4202233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4202620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4202812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4202832Z 
2022-12-01T11:08:36.4202944Z Running tests...
2022-12-01T11:08:36.4203207Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4203521Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4203780Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4204543Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2022-12-01T11:08:36.4204566Z 
2022-12-01T11:08:36.4204830Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4204943Z Ran 1 test in 1.600s
2022-12-01T11:08:36.4204963Z 
2022-12-01T11:08:36.4205070Z OK (skipped=1)
2022-12-01T11:08:36.4205089Z 
2022-12-01T11:08:36.4205213Z Generating XML reports...
2022-12-01T11:08:36.4205660Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110414.xml
2022-12-01T11:08:36.4206032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4206212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4206595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4206823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4206844Z 
2022-12-01T11:08:36.4206957Z Running tests...
2022-12-01T11:08:36.4207225Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4207536Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4207831Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4208577Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.674s)
2022-12-01T11:08:36.4208598Z 
2022-12-01T11:08:36.4208865Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4208982Z Ran 1 test in 1.674s
2022-12-01T11:08:36.4209001Z 
2022-12-01T11:08:36.4209114Z OK (skipped=1)
2022-12-01T11:08:36.4209133Z 
2022-12-01T11:08:36.4209238Z Generating XML reports...
2022-12-01T11:08:36.4209685Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110417.xml
2022-12-01T11:08:36.4210056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4210233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4210617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4210810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4210829Z 
2022-12-01T11:08:36.4210937Z Running tests...
2022-12-01T11:08:36.4211202Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4211581Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4211873Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4212096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102725
2022-12-01T11:08:36.4212316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102726
2022-12-01T11:08:36.4212692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4212872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4213252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4213442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4213819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4213999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4214354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4214543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4214792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4215037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4215443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4215846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4216134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4216370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4216501Z skip: Need at least 4 CUDA devices (3.950s)
2022-12-01T11:08:36.4216543Z 
2022-12-01T11:08:36.4216797Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4216911Z Ran 1 test in 3.950s
2022-12-01T11:08:36.4216930Z 
2022-12-01T11:08:36.4217038Z OK (skipped=1)
2022-12-01T11:08:36.4217058Z 
2022-12-01T11:08:36.4217183Z Generating XML reports...
2022-12-01T11:08:36.4217635Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110421.xml
2022-12-01T11:08:36.4218009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4218188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4218581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4218755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4218795Z 
2022-12-01T11:08:36.4218887Z Running tests...
2022-12-01T11:08:36.4219151Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4219464Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4219791Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4220017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103046
2022-12-01T11:08:36.4220239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103047
2022-12-01T11:08:36.4220614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4220905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4221270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4221463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4221832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4222006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4222388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4222580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4222828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4223080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4223464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4223868Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4224100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4224328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4224480Z skip: Need at least 4 CUDA devices (4.094s)
2022-12-01T11:08:36.4224500Z 
2022-12-01T11:08:36.4224769Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4224883Z Ran 1 test in 4.094s
2022-12-01T11:08:36.4224902Z 
2022-12-01T11:08:36.4225015Z OK (skipped=1)
2022-12-01T11:08:36.4225034Z 
2022-12-01T11:08:36.4225158Z Generating XML reports...
2022-12-01T11:08:36.4225638Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110428.xml
2022-12-01T11:08:36.4226024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4226202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4226581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4226775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4226794Z 
2022-12-01T11:08:36.4226904Z Running tests...
2022-12-01T11:08:36.4227171Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4227486Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4227783Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4228510Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.651s)
2022-12-01T11:08:36.4228552Z 
2022-12-01T11:08:36.4228801Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4228916Z Ran 1 test in 1.651s
2022-12-01T11:08:36.4228935Z 
2022-12-01T11:08:36.4229043Z OK (skipped=1)
2022-12-01T11:08:36.4229062Z 
2022-12-01T11:08:36.4229186Z Generating XML reports...
2022-12-01T11:08:36.4229995Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110434.xml
2022-12-01T11:08:36.4230482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4230663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4231046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4231219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4231260Z 
2022-12-01T11:08:36.4231351Z Running tests...
2022-12-01T11:08:36.4231617Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4231933Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4232202Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4232425Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103444
2022-12-01T11:08:36.4232651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103445
2022-12-01T11:08:36.4233032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4233211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4233575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4233773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4234139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4234312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4234687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4234880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4235194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4235452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4235838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4236239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4236471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4236716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4236943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4237180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4237594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4237991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4238335Z STAGE:2022-12-01 11:04:42 103445:103445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4238665Z STAGE:2022-12-01 11:04:42 103444:103444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4238887Z [1669892684.119431] [7ff5451e42d4:103445:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4239133Z [1669892684.119431] [7ff5451e42d4:103445:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4239369Z [1669892684.101869] [7ff5451e42d4:103444:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4239671Z [1669892684.101869] [7ff5451e42d4:103444:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4240234Z STAGE:2022-12-01 11:04:44 103445:103445 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:04:44 103444:103444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4240256Z 
2022-12-01T11:08:36.4240586Z STAGE:2022-12-01 11:04:44 103445:103445 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4240911Z STAGE:2022-12-01 11:04:44 103444:103444 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4241251Z STAGE:2022-12-01 11:04:44 103445:103445 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4241588Z STAGE:2022-12-01 11:04:44 103444:103444 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4241675Z ok (7.311s)
2022-12-01T11:08:36.4241694Z 
2022-12-01T11:08:36.4241966Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4242081Z Ran 1 test in 7.312s
2022-12-01T11:08:36.4242100Z 
2022-12-01T11:08:36.4242193Z OK
2022-12-01T11:08:36.4242212Z 
2022-12-01T11:08:36.4242335Z Generating XML reports...
2022-12-01T11:08:36.4242791Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110438.xml
2022-12-01T11:08:36.4243165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4243342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4243724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4243898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4243921Z 
2022-12-01T11:08:36.4244031Z Running tests...
2022-12-01T11:08:36.4244346Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4244671Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4244935Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4245153Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103785
2022-12-01T11:08:36.4245374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103786
2022-12-01T11:08:36.4245750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4245909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4246293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4246492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4246861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4247035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4247409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4247597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4247846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4248089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4248474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4248939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4249170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4249409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4249638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4249880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4250280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4250622Z STAGE:2022-12-01 11:04:51 103785:103785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4251017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4251345Z STAGE:2022-12-01 11:04:51 103786:103786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4251585Z [1669892693.408262] [7ff5451e42d4:103785:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4251828Z [1669892693.408262] [7ff5451e42d4:103785:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4252053Z [1669892693.461149] [7ff5451e42d4:103786:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4252288Z [1669892693.461149] [7ff5451e42d4:103786:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4252846Z STAGE:2022-12-01 11:04:53 103785:103785 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:04:53 103786:103786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4252871Z 
2022-12-01T11:08:36.4253261Z STAGE:2022-12-01 11:04:53 103786:103786 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4253601Z STAGE:2022-12-01 11:04:53 103785:103785 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4253941Z STAGE:2022-12-01 11:04:54 103786:103786 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4254282Z STAGE:2022-12-01 11:04:54 103785:103785 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4254365Z ok (7.085s)
2022-12-01T11:08:36.4254384Z 
2022-12-01T11:08:36.4254651Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4254767Z Ran 1 test in 7.085s
2022-12-01T11:08:36.4254787Z 
2022-12-01T11:08:36.4254880Z OK
2022-12-01T11:08:36.4254899Z 
2022-12-01T11:08:36.4255022Z Generating XML reports...
2022-12-01T11:08:36.4255475Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110447.xml
2022-12-01T11:08:36.4255856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4256036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4256415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4256591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4256610Z 
2022-12-01T11:08:36.4256719Z Running tests...
2022-12-01T11:08:36.4256982Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4257292Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4257565Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4257864Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104153
2022-12-01T11:08:36.4258086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104154
2022-12-01T11:08:36.4258467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4258625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4259007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4259197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4259562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4259736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4260112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4260309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4260557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4260800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4261189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4261588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4261819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4262062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4262290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4262571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4262980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4263378Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4263719Z STAGE:2022-12-01 11:05:01 104153:104153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4264031Z STAGE:2022-12-01 11:05:01 104154:104154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4264272Z [1669892702.968178] [7ff5451e42d4:104153:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4264516Z [1669892702.968178] [7ff5451e42d4:104153:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4264864Z STAGE:2022-12-01 11:05:03 104153:104153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4265098Z [1669892702.944838] [7ff5451e42d4:104154:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4265330Z [1669892702.944838] [7ff5451e42d4:104154:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4265665Z STAGE:2022-12-01 11:05:03 104154:104154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4265998Z STAGE:2022-12-01 11:05:03 104153:104153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4266332Z STAGE:2022-12-01 11:05:03 104153:104153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4266641Z STAGE:2022-12-01 11:05:03 104154:104154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4267042Z STAGE:2022-12-01 11:05:03 104154:104154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4267148Z ok (7.205s)
2022-12-01T11:08:36.4267167Z 
2022-12-01T11:08:36.4267436Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4267550Z Ran 1 test in 7.205s
2022-12-01T11:08:36.4267570Z 
2022-12-01T11:08:36.4267662Z OK
2022-12-01T11:08:36.4267681Z 
2022-12-01T11:08:36.4267803Z Generating XML reports...
2022-12-01T11:08:36.4268252Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110457.xml
2022-12-01T11:08:36.4268629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4268788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4269169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4269604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4269626Z 
2022-12-01T11:08:36.4269745Z Running tests...
2022-12-01T11:08:36.4270019Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4270333Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4270601Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4270859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104539
2022-12-01T11:08:36.4271067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104540
2022-12-01T11:08:36.4271447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4271623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4272087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4272289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4272658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4272833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4273209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4273397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4273627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4273871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4274287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4274685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4274916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4275157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:08:36.4275380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4275617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:08:36.4276020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4276344Z STAGE:2022-12-01 11:05:10 104539:104539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4276829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:08:36.4277164Z STAGE:2022-12-01 11:05:10 104540:104540 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4277404Z [1669892712.321843] [7ff5451e42d4:104539:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4277647Z [1669892712.321843] [7ff5451e42d4:104539:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4277875Z [1669892712.297201] [7ff5451e42d4:104540:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4278107Z [1669892712.297201] [7ff5451e42d4:104540:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4278676Z STAGE:2022-12-01 11:05:12 104539:104539 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:05:12 104540:104540 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4278698Z 
2022-12-01T11:08:36.4279036Z STAGE:2022-12-01 11:05:12 104540:104540 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4279363Z STAGE:2022-12-01 11:05:12 104539:104539 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4279682Z STAGE:2022-12-01 11:05:12 104540:104540 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4280014Z STAGE:2022-12-01 11:05:12 104539:104539 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4280117Z ok (7.229s)
2022-12-01T11:08:36.4280136Z 
2022-12-01T11:08:36.4280402Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4280518Z Ran 1 test in 7.229s
2022-12-01T11:08:36.4280538Z 
2022-12-01T11:08:36.4280634Z OK
2022-12-01T11:08:36.4280653Z 
2022-12-01T11:08:36.4280778Z Generating XML reports...
2022-12-01T11:08:36.4281277Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110506.xml
2022-12-01T11:08:36.4281667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4281828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4282209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4282403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4282422Z 
2022-12-01T11:08:36.4282531Z Running tests...
2022-12-01T11:08:36.4282795Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4283113Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4283377Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4283602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104838
2022-12-01T11:08:36.4283803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104841
2022-12-01T11:08:36.4284184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4284366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4284751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4284944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4285309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4285538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4285925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4286112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4286341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4286586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4286989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4287386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4287619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4287852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4288017Z skip: Skipped due to small world size. (4.060s)
2022-12-01T11:08:36.4288037Z 
2022-12-01T11:08:36.4288309Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4288404Z Ran 1 test in 4.060s
2022-12-01T11:08:36.4288441Z 
2022-12-01T11:08:36.4288532Z OK (skipped=1)
2022-12-01T11:08:36.4288551Z 
2022-12-01T11:08:36.4288674Z Generating XML reports...
2022-12-01T11:08:36.4289124Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110516.xml
2022-12-01T11:08:36.4289503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4289677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4290059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4290305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4290326Z 
2022-12-01T11:08:36.4290439Z Running tests...
2022-12-01T11:08:36.4290689Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4291001Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4291257Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4291479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105160
2022-12-01T11:08:36.4291703Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105161
2022-12-01T11:08:36.4292075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4292250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4292638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4292815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4293189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4293364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4293739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4293929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4294175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4294420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4294886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4295288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4295499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4295728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4295885Z skip: Skipped due to small world size. (3.986s)
2022-12-01T11:08:36.4295905Z 
2022-12-01T11:08:36.4296174Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4296287Z Ran 1 test in 3.986s
2022-12-01T11:08:36.4296306Z 
2022-12-01T11:08:36.4296412Z OK (skipped=1)
2022-12-01T11:08:36.4296431Z 
2022-12-01T11:08:36.4296555Z Generating XML reports...
2022-12-01T11:08:36.4297007Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110522.xml
2022-12-01T11:08:36.4297386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4297547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4297928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4298119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4298138Z 
2022-12-01T11:08:36.4298248Z Running tests...
2022-12-01T11:08:36.4298514Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4298827Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4299090Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4299316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105408
2022-12-01T11:08:36.4299571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105409
2022-12-01T11:08:36.4299967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4300145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4300524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4300716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4301082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4301257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4301628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4301825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4302054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4302295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4302699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4303097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4303327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4303557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4303721Z skip: Skipped due to small world size. (3.933s)
2022-12-01T11:08:36.4303792Z 
2022-12-01T11:08:36.4304070Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4304183Z Ran 1 test in 3.933s
2022-12-01T11:08:36.4304202Z 
2022-12-01T11:08:36.4304291Z OK (skipped=1)
2022-12-01T11:08:36.4304311Z 
2022-12-01T11:08:36.4304435Z Generating XML reports...
2022-12-01T11:08:36.4304885Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110529.xml
2022-12-01T11:08:36.4305261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4305438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4305819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4306012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4306032Z 
2022-12-01T11:08:36.4306145Z Running tests...
2022-12-01T11:08:36.4306398Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4306712Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4306972Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4307194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105732
2022-12-01T11:08:36.4307410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105733
2022-12-01T11:08:36.4307779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4307956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4308335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4308534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4308935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4309119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4309746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4309945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4310195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4310446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4310857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4311256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4311494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4311705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4311867Z skip: Skipped due to small world size. (4.081s)
2022-12-01T11:08:36.4311888Z 
2022-12-01T11:08:36.4312157Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4312270Z Ran 1 test in 4.081s
2022-12-01T11:08:36.4312289Z 
2022-12-01T11:08:36.4312397Z OK (skipped=1)
2022-12-01T11:08:36.4312416Z 
2022-12-01T11:08:36.4312540Z Generating XML reports...
2022-12-01T11:08:36.4312990Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110535.xml
2022-12-01T11:08:36.4313360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4313610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4314001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4314195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4314215Z 
2022-12-01T11:08:36.4314327Z Running tests...
2022-12-01T11:08:36.4314595Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4314908Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4315159Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4315381Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105999
2022-12-01T11:08:36.4315605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106000
2022-12-01T11:08:36.4315966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4316144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4316525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4316716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4317082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4317260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4317636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4317825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4318053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4318364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4318779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4319177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4319406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4319746Z STAGE:2022-12-01 11:05:45 106000:106000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4319972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4320303Z STAGE:2022-12-01 11:05:45 105999:105999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4320548Z [1669892746.935759] [7ff5451e42d4:106000:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4320771Z [1669892746.935759] [7ff5451e42d4:106000:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4321117Z STAGE:2022-12-01 11:05:47 106000:106000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4321345Z [1669892746.956384] [7ff5451e42d4:105999:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4321576Z [1669892746.956384] [7ff5451e42d4:105999:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4321920Z STAGE:2022-12-01 11:05:47 105999:105999 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4322253Z STAGE:2022-12-01 11:05:47 106000:106000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4322665Z STAGE:2022-12-01 11:05:47 105999:105999 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4323005Z STAGE:2022-12-01 11:05:47 106000:106000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4323349Z STAGE:2022-12-01 11:05:47 105999:105999 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4323439Z ok (7.265s)
2022-12-01T11:08:36.4323480Z 
2022-12-01T11:08:36.4323729Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4323938Z Ran 1 test in 7.266s
2022-12-01T11:08:36.4323959Z 
2022-12-01T11:08:36.4324055Z OK
2022-12-01T11:08:36.4324073Z 
2022-12-01T11:08:36.4324201Z Generating XML reports...
2022-12-01T11:08:36.4324656Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110541.xml
2022-12-01T11:08:36.4325032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4325220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4325607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4325783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4325821Z 
2022-12-01T11:08:36.4325911Z Running tests...
2022-12-01T11:08:36.4326174Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4326487Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4326733Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4326950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106422
2022-12-01T11:08:36.4327167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106423
2022-12-01T11:08:36.4327595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4327759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4328142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4328330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4328695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4328870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4329245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4329431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4329678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4329930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4330316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4330715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4330943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4331286Z STAGE:2022-12-01 11:05:54 106423:106423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4331511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4331840Z STAGE:2022-12-01 11:05:54 106422:106422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4332139Z [1669892756.338092] [7ff5451e42d4:106423:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4332383Z [1669892756.338092] [7ff5451e42d4:106423:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4332615Z [1669892756.378307] [7ff5451e42d4:106422:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4332833Z [1669892756.378307] [7ff5451e42d4:106422:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4333397Z STAGE:2022-12-01 11:05:56 106423:106423 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:05:56 106422:106422 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4333437Z 
2022-12-01T11:08:36.4333753Z STAGE:2022-12-01 11:05:56 106423:106423 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4334086Z STAGE:2022-12-01 11:05:56 106422:106422 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4334635Z STAGE:2022-12-01 11:05:56 106423:106423 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:05:56 106422:106422 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4334655Z 
2022-12-01T11:08:36.4334756Z ok (7.130s)
2022-12-01T11:08:36.4334775Z 
2022-12-01T11:08:36.4335042Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4335155Z Ran 1 test in 7.130s
2022-12-01T11:08:36.4335175Z 
2022-12-01T11:08:36.4335267Z OK
2022-12-01T11:08:36.4335286Z 
2022-12-01T11:08:36.4335411Z Generating XML reports...
2022-12-01T11:08:36.4335863Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110550.xml
2022-12-01T11:08:36.4336220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4336453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4336846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4337037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4337057Z 
2022-12-01T11:08:36.4337166Z Running tests...
2022-12-01T11:08:36.4337431Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4337742Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4338021Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-12-01T11:08:36.4338041Z 
2022-12-01T11:08:36.4338307Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4338402Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4338426Z 
2022-12-01T11:08:36.4338534Z OK (skipped=1)
2022-12-01T11:08:36.4338553Z 
2022-12-01T11:08:36.4338679Z Generating XML reports...
2022-12-01T11:08:36.4339125Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110600.xml
2022-12-01T11:08:36.4339496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4339673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4340056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4340248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4340268Z 
2022-12-01T11:08:36.4340359Z Running tests...
2022-12-01T11:08:36.4340622Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4340935Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4341251Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4341477Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106732
2022-12-01T11:08:36.4341697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106733
2022-12-01T11:08:36.4342076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4342253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4342636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4342813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4343181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4343363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4343740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4343928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4344175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4344422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4344823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4345206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4345439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4345827Z STAGE:2022-12-01 11:06:05 106733:106733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4346062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4346397Z STAGE:2022-12-01 11:06:05 106732:106732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4346635Z [1669892767.434681] [7ff5451e42d4:106732:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4346878Z [1669892767.434681] [7ff5451e42d4:106732:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4347112Z [1669892767.392637] [7ff5451e42d4:106733:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4347352Z [1669892767.392637] [7ff5451e42d4:106733:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4347921Z STAGE:2022-12-01 11:06:07 106732:106732 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:06:07 106733:106733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4347946Z 
2022-12-01T11:08:36.4348262Z STAGE:2022-12-01 11:06:07 106733:106733 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4348590Z STAGE:2022-12-01 11:06:08 106732:106732 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4348932Z STAGE:2022-12-01 11:06:08 106733:106733 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4349278Z STAGE:2022-12-01 11:06:08 106732:106732 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4349612Z ok (7.022s)
2022-12-01T11:08:36.4349634Z 
2022-12-01T11:08:36.4349918Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4350034Z Ran 1 test in 7.023s
2022-12-01T11:08:36.4350130Z 
2022-12-01T11:08:36.4350230Z OK
2022-12-01T11:08:36.4350249Z 
2022-12-01T11:08:36.4350381Z Generating XML reports...
2022-12-01T11:08:36.4350820Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110602.xml
2022-12-01T11:08:36.4351194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4351371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4351753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4351943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4351962Z 
2022-12-01T11:08:36.4352071Z Running tests...
2022-12-01T11:08:36.4352340Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4352654Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4352957Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-12-01T11:08:36.4352978Z 
2022-12-01T11:08:36.4353226Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4353338Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4353357Z 
2022-12-01T11:08:36.4353464Z OK (skipped=1)
2022-12-01T11:08:36.4353484Z 
2022-12-01T11:08:36.4353608Z Generating XML reports...
2022-12-01T11:08:36.4354057Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110611.xml
2022-12-01T11:08:36.4354431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4354605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4354986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4355248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4355270Z 
2022-12-01T11:08:36.4355367Z Running tests...
2022-12-01T11:08:36.4355636Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4355947Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4356223Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-12-01T11:08:36.4356243Z 
2022-12-01T11:08:36.4356504Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4356618Z Ran 1 test in 0.003s
2022-12-01T11:08:36.4356637Z 
2022-12-01T11:08:36.4356747Z OK (skipped=1)
2022-12-01T11:08:36.4356766Z 
2022-12-01T11:08:36.4356888Z Generating XML reports...
2022-12-01T11:08:36.4357320Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110613.xml
2022-12-01T11:08:36.4357696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4357871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4358252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4358447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4358467Z 
2022-12-01T11:08:36.4358575Z Running tests...
2022-12-01T11:08:36.4358840Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4359154Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4359402Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4359665Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107060
2022-12-01T11:08:36.4359888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107061
2022-12-01T11:08:36.4360266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4360441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4360821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4361014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4361378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4361551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4361906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4362107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4362354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4362601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4363002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4363402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4363633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4363976Z STAGE:2022-12-01 11:06:19 107061:107061 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4364201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4364632Z STAGE:2022-12-01 11:06:19 107060:107060 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4364880Z [1669892780.859097] [7ff5451e42d4:107061:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4365121Z [1669892780.859097] [7ff5451e42d4:107061:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4365354Z [1669892780.884059] [7ff5451e42d4:107060:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4365589Z [1669892780.884059] [7ff5451e42d4:107060:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4366149Z STAGE:2022-12-01 11:06:21 107061:107061 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:06:21 107060:107060 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4366173Z 
2022-12-01T11:08:36.4366512Z STAGE:2022-12-01 11:06:21 107061:107061 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4366843Z STAGE:2022-12-01 11:06:21 107060:107060 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4367182Z STAGE:2022-12-01 11:06:21 107061:107061 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4367522Z STAGE:2022-12-01 11:06:21 107060:107060 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4367607Z ok (7.176s)
2022-12-01T11:08:36.4367643Z 
2022-12-01T11:08:36.4367893Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4368007Z Ran 1 test in 7.177s
2022-12-01T11:08:36.4368026Z 
2022-12-01T11:08:36.4368120Z OK
2022-12-01T11:08:36.4368138Z 
2022-12-01T11:08:36.4368263Z Generating XML reports...
2022-12-01T11:08:36.4368782Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110615.xml
2022-12-01T11:08:36.4369158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4369336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4369720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4369895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4369915Z 
2022-12-01T11:08:36.4370024Z Running tests...
2022-12-01T11:08:36.4370290Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4370605Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4370915Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T11:08:36.4370940Z 
2022-12-01T11:08:36.4371207Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4371322Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4371341Z 
2022-12-01T11:08:36.4371451Z OK (skipped=1)
2022-12-01T11:08:36.4371471Z 
2022-12-01T11:08:36.4371594Z Generating XML reports...
2022-12-01T11:08:36.4372025Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110624.xml
2022-12-01T11:08:36.4372401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4372574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4372955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4373149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4373172Z 
2022-12-01T11:08:36.4373279Z Running tests...
2022-12-01T11:08:36.4373593Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4373916Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4374164Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T11:08:36.4374203Z 
2022-12-01T11:08:36.4374445Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4374555Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4374574Z 
2022-12-01T11:08:36.4374678Z OK (skipped=1)
2022-12-01T11:08:36.4374697Z 
2022-12-01T11:08:36.4374819Z Generating XML reports...
2022-12-01T11:08:36.4375266Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110626.xml
2022-12-01T11:08:36.4375637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4375821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4376205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4376378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4376399Z 
2022-12-01T11:08:36.4376505Z Running tests...
2022-12-01T11:08:36.4376768Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4377079Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4377335Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4377559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107564
2022-12-01T11:08:36.4377779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107565
2022-12-01T11:08:36.4378226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4378387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4378769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4378961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4379326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4379500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4379873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4380064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4380314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4380566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4380952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4381351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4381584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4381923Z STAGE:2022-12-01 11:06:32 107564:107564 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4382148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4382478Z STAGE:2022-12-01 11:06:32 107565:107565 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4382769Z [1669892794.016715] [7ff5451e42d4:107565:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4383019Z [1669892794.016715] [7ff5451e42d4:107565:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4383250Z [1669892794.015024] [7ff5451e42d4:107564:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4383469Z [1669892794.015024] [7ff5451e42d4:107564:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4384031Z STAGE:2022-12-01 11:06:34 107565:107565 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:06:34 107564:107564 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4384052Z 
2022-12-01T11:08:36.4384386Z STAGE:2022-12-01 11:06:34 107565:107565 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4384721Z STAGE:2022-12-01 11:06:34 107564:107564 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4385062Z STAGE:2022-12-01 11:06:34 107565:107565 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4385401Z STAGE:2022-12-01 11:06:34 107564:107564 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4385505Z ok (6.997s)
2022-12-01T11:08:36.4385524Z 
2022-12-01T11:08:36.4385789Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4385901Z Ran 1 test in 6.997s
2022-12-01T11:08:36.4385920Z 
2022-12-01T11:08:36.4386011Z OK
2022-12-01T11:08:36.4386030Z 
2022-12-01T11:08:36.4386136Z Generating XML reports...
2022-12-01T11:08:36.4386590Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110628.xml
2022-12-01T11:08:36.4386963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4387218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4387604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4387795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4387815Z 
2022-12-01T11:08:36.4387924Z Running tests...
2022-12-01T11:08:36.4388189Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4388483Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4388746Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4388766Z 
2022-12-01T11:08:36.4389031Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4389143Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4389166Z 
2022-12-01T11:08:36.4389274Z OK (skipped=1)
2022-12-01T11:08:36.4389293Z 
2022-12-01T11:08:36.4389657Z Generating XML reports...
2022-12-01T11:08:36.4390127Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110637.xml
2022-12-01T11:08:36.4390501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4390676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4391042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4391239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4391258Z 
2022-12-01T11:08:36.4391371Z Running tests...
2022-12-01T11:08:36.4391639Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4391952Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4392305Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4392326Z 
2022-12-01T11:08:36.4392600Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4392715Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4392734Z 
2022-12-01T11:08:36.4392841Z OK (skipped=1)
2022-12-01T11:08:36.4392860Z 
2022-12-01T11:08:36.4392965Z Generating XML reports...
2022-12-01T11:08:36.4393413Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110639.xml
2022-12-01T11:08:36.4393787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4393964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4394342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4394543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4394563Z 
2022-12-01T11:08:36.4394673Z Running tests...
2022-12-01T11:08:36.4394943Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4395239Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4395512Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4395532Z 
2022-12-01T11:08:36.4395795Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4395907Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4395926Z 
2022-12-01T11:08:36.4396033Z OK (skipped=1)
2022-12-01T11:08:36.4396052Z 
2022-12-01T11:08:36.4396175Z Generating XML reports...
2022-12-01T11:08:36.4396617Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110641.xml
2022-12-01T11:08:36.4397071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4397248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4397608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4397804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4397824Z 
2022-12-01T11:08:36.4397932Z Running tests...
2022-12-01T11:08:36.4398197Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4398507Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4398759Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T11:08:36.4398782Z 
2022-12-01T11:08:36.4399047Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4399165Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4399184Z 
2022-12-01T11:08:36.4399289Z OK (skipped=1)
2022-12-01T11:08:36.4399309Z 
2022-12-01T11:08:36.4399413Z Generating XML reports...
2022-12-01T11:08:36.4399862Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110644.xml
2022-12-01T11:08:36.4400232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4400410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4400790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4400980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4401000Z 
2022-12-01T11:08:36.4401109Z Running tests...
2022-12-01T11:08:36.4401379Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4401745Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4402001Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T11:08:36.4402021Z 
2022-12-01T11:08:36.4402285Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4402396Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4402415Z 
2022-12-01T11:08:36.4402522Z OK (skipped=1)
2022-12-01T11:08:36.4402541Z 
2022-12-01T11:08:36.4402666Z Generating XML reports...
2022-12-01T11:08:36.4403115Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110646.xml
2022-12-01T11:08:36.4403488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4403672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4404041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4404233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4404252Z 
2022-12-01T11:08:36.4404359Z Running tests...
2022-12-01T11:08:36.4404623Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4404933Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4405207Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4405228Z 
2022-12-01T11:08:36.4405487Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4405601Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4405620Z 
2022-12-01T11:08:36.4405727Z OK (skipped=1)
2022-12-01T11:08:36.4405795Z 
2022-12-01T11:08:36.4405909Z Generating XML reports...
2022-12-01T11:08:36.4406359Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110648.xml
2022-12-01T11:08:36.4406730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4406904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4407285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4407477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4407497Z 
2022-12-01T11:08:36.4407604Z Running tests...
2022-12-01T11:08:36.4407871Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4408182Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4408441Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: CPU tensor ops not supported by UCP TL (0.002s)
2022-12-01T11:08:36.4408461Z 
2022-12-01T11:08:36.4408725Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4408838Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4408856Z 
2022-12-01T11:08:36.4408964Z OK (skipped=1)
2022-12-01T11:08:36.4408983Z 
2022-12-01T11:08:36.4409106Z Generating XML reports...
2022-12-01T11:08:36.4409551Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110650.xml
2022-12-01T11:08:36.4409928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4410102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4410480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4410656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4410724Z 
2022-12-01T11:08:36.4410837Z Running tests...
2022-12-01T11:08:36.4411103Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4411413Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4411799Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:08:36.4411819Z 
2022-12-01T11:08:36.4412081Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4412193Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4412212Z 
2022-12-01T11:08:36.4412320Z OK (skipped=1)
2022-12-01T11:08:36.4412339Z 
2022-12-01T11:08:36.4412445Z Generating XML reports...
2022-12-01T11:08:36.4412892Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110652.xml
2022-12-01T11:08:36.4413269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4413445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4413821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4414012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4414032Z 
2022-12-01T11:08:36.4414141Z Running tests...
2022-12-01T11:08:36.4414406Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4414716Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4414942Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4415168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108606
2022-12-01T11:08:36.4415452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108607
2022-12-01T11:08:36.4415829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4416047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4416417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4416592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4416978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4417152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4417534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4417727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4417973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4418218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4418618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4419018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4419251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4419481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4419703Z [1669892819.692333] [7ff5451e42d4:108607:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4419999Z [1669892819.692333] [7ff5451e42d4:108607:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4420239Z [1669892819.692094] [7ff5451e42d4:108606:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4420473Z [1669892819.692094] [7ff5451e42d4:108606:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4420576Z ok (6.070s)
2022-12-01T11:08:36.4420596Z 
2022-12-01T11:08:36.4420869Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4420983Z Ran 1 test in 6.070s
2022-12-01T11:08:36.4421003Z 
2022-12-01T11:08:36.4421096Z OK
2022-12-01T11:08:36.4421114Z 
2022-12-01T11:08:36.4421238Z Generating XML reports...
2022-12-01T11:08:36.4421671Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110654.xml
2022-12-01T11:08:36.4422059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4422238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4422621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4422808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4422828Z 
2022-12-01T11:08:36.4422936Z Running tests...
2022-12-01T11:08:36.4423202Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4423514Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4423861Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-12-01T11:08:36.4423883Z 
2022-12-01T11:08:36.4424130Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4424301Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4424323Z 
2022-12-01T11:08:36.4424432Z OK (skipped=1)
2022-12-01T11:08:36.4424450Z 
2022-12-01T11:08:36.4424574Z Generating XML reports...
2022-12-01T11:08:36.4425027Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110702.xml
2022-12-01T11:08:36.4425398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4425576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4425959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4426150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4426170Z 
2022-12-01T11:08:36.4426259Z Running tests...
2022-12-01T11:08:36.4426522Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4426845Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4427155Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-12-01T11:08:36.4427176Z 
2022-12-01T11:08:36.4427436Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4427551Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4427570Z 
2022-12-01T11:08:36.4427681Z OK (skipped=1)
2022-12-01T11:08:36.4427699Z 
2022-12-01T11:08:36.4427823Z Generating XML reports...
2022-12-01T11:08:36.4428268Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110704.xml
2022-12-01T11:08:36.4428620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4428804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4429229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4429659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4429681Z 
2022-12-01T11:08:36.4429796Z Running tests...
2022-12-01T11:08:36.4430073Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4430387Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4430690Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: ucc does not support send/recv from any source (0.002s)
2022-12-01T11:08:36.4430710Z 
2022-12-01T11:08:36.4430974Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4431068Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4431087Z 
2022-12-01T11:08:36.4431199Z OK (skipped=1)
2022-12-01T11:08:36.4431218Z 
2022-12-01T11:08:36.4431341Z Generating XML reports...
2022-12-01T11:08:36.4431788Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110706.xml
2022-12-01T11:08:36.4432163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4432344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4432727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4432920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4432939Z 
2022-12-01T11:08:36.4433030Z Running tests...
2022-12-01T11:08:36.4433293Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4433603Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4433979Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4434202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109241
2022-12-01T11:08:36.4434421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109243
2022-12-01T11:08:36.4434799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4434977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4435356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4435528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4435899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4436082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4436459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4436648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4436896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4437143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4437555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4437937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4438168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4438573Z STAGE:2022-12-01 11:07:12 109241:109241 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4438807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4439142Z STAGE:2022-12-01 11:07:12 109243:109243 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4439378Z [1669892834.400610] [7ff5451e42d4:109243:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4439620Z [1669892834.400610] [7ff5451e42d4:109243:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4439968Z STAGE:2022-12-01 11:07:14 109243:109243 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4440197Z [1669892834.402359] [7ff5451e42d4:109241:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4440432Z [1669892834.402359] [7ff5451e42d4:109241:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4440764Z STAGE:2022-12-01 11:07:14 109241:109241 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4440867Z ok (7.104s)
2022-12-01T11:08:36.4440887Z 
2022-12-01T11:08:36.4441155Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4441268Z Ran 1 test in 7.104s
2022-12-01T11:08:36.4441287Z 
2022-12-01T11:08:36.4441380Z OK
2022-12-01T11:08:36.4441399Z 
2022-12-01T11:08:36.4441522Z Generating XML reports...
2022-12-01T11:08:36.4441972Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110708.xml
2022-12-01T11:08:36.4442347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4442507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4442954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4443146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4443166Z 
2022-12-01T11:08:36.4443275Z Running tests...
2022-12-01T11:08:36.4443538Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4443847Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4444085Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T11:08:36.4444105Z 
2022-12-01T11:08:36.4444365Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4444480Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4444499Z 
2022-12-01T11:08:36.4444588Z OK (skipped=1)
2022-12-01T11:08:36.4444607Z 
2022-12-01T11:08:36.4444733Z Generating XML reports...
2022-12-01T11:08:36.4445188Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110718.xml
2022-12-01T11:08:36.4445562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4445737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4446115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4446309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4446329Z 
2022-12-01T11:08:36.4446436Z Running tests...
2022-12-01T11:08:36.4446703Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4446998Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4447266Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T11:08:36.4447290Z 
2022-12-01T11:08:36.4447621Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4447739Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4447758Z 
2022-12-01T11:08:36.4447866Z OK (skipped=1)
2022-12-01T11:08:36.4447885Z 
2022-12-01T11:08:36.4448010Z Generating XML reports...
2022-12-01T11:08:36.4448458Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110720.xml
2022-12-01T11:08:36.4448830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4448988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4449369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4449564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4449587Z 
2022-12-01T11:08:36.4449694Z Running tests...
2022-12-01T11:08:36.4449965Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4450279Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4450540Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T11:08:36.4450560Z 
2022-12-01T11:08:36.4450818Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4450931Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4450950Z 
2022-12-01T11:08:36.4451039Z OK (skipped=1)
2022-12-01T11:08:36.4451058Z 
2022-12-01T11:08:36.4451186Z Generating XML reports...
2022-12-01T11:08:36.4451634Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110722.xml
2022-12-01T11:08:36.4452006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4452263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4452649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4452846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4452867Z 
2022-12-01T11:08:36.4452978Z Running tests...
2022-12-01T11:08:36.4453241Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4453535Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4453808Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4454029Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109927
2022-12-01T11:08:36.4454247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109928
2022-12-01T11:08:36.4454629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4454807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4455188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4455381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4455731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4455908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4456284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4456476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4456732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4457254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4458038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4458806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4459221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4459812Z STAGE:2022-12-01 11:07:28 109927:109927 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4460124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4460477Z STAGE:2022-12-01 11:07:28 109928:109928 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4460727Z [1669892850.074978] [7ff5451e42d4:109928:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4460971Z [1669892850.074978] [7ff5451e42d4:109928:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4461319Z STAGE:2022-12-01 11:07:30 109928:109928 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4461558Z [1669892850.095992] [7ff5451e42d4:109927:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4461816Z [1669892850.095992] [7ff5451e42d4:109927:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4462466Z STAGE:2022-12-01 11:07:30 109927:109927 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4462567Z ok (7.136s)
2022-12-01T11:08:36.4462609Z 
2022-12-01T11:08:36.4462977Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4463103Z Ran 1 test in 7.136s
2022-12-01T11:08:36.4463125Z 
2022-12-01T11:08:36.4463218Z OK
2022-12-01T11:08:36.4463237Z 
2022-12-01T11:08:36.4463361Z Generating XML reports...
2022-12-01T11:08:36.4463815Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110724.xml
2022-12-01T11:08:36.4464188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4464374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4464757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4464933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4464953Z 
2022-12-01T11:08:36.4465059Z Running tests...
2022-12-01T11:08:36.4465324Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4465645Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4465907Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4466126Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110332
2022-12-01T11:08:36.4466345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110333
2022-12-01T11:08:36.4466724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4466885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4467271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4467464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4467890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4468075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4468459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4468650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4468898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4469143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4470173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4470617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4470858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4471128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4471366Z [1669892859.212078] [7ff5451e42d4:110333:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4471610Z [1669892859.212078] [7ff5451e42d4:110333:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4471842Z [1669892859.213129] [7ff5451e42d4:110332:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4492664Z [1669892859.213129] [7ff5451e42d4:110332:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4492825Z ok (5.958s)
2022-12-01T11:08:36.4492847Z 
2022-12-01T11:08:36.4493170Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4493439Z Ran 1 test in 5.958s
2022-12-01T11:08:36.4493464Z 
2022-12-01T11:08:36.4493558Z OK
2022-12-01T11:08:36.4493577Z 
2022-12-01T11:08:36.4493713Z Generating XML reports...
2022-12-01T11:08:36.4494190Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110734.xml
2022-12-01T11:08:36.4494570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4494749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4495137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4495331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4495352Z 
2022-12-01T11:08:36.4495461Z Running tests...
2022-12-01T11:08:36.4495715Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4496040Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4496330Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4496555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110663
2022-12-01T11:08:36.4496774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110664
2022-12-01T11:08:36.4497150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4497328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4497706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4497882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4498336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4498530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4498915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4499105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4499351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4499597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4500003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4500407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4500629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4500976Z STAGE:2022-12-01 11:07:46 110664:110664 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4501206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4501548Z STAGE:2022-12-01 11:07:46 110663:110663 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4501785Z [1669892867.783990] [7ff5451e42d4:110663:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4502026Z [1669892867.783990] [7ff5451e42d4:110663:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4502369Z STAGE:2022-12-01 11:07:48 110663:110663 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4502672Z [1669892867.846197] [7ff5451e42d4:110664:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4502909Z [1669892867.846197] [7ff5451e42d4:110664:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4503241Z STAGE:2022-12-01 11:07:48 110664:110664 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4503344Z ok (7.237s)
2022-12-01T11:08:36.4503365Z 
2022-12-01T11:08:36.4503630Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4503736Z Ran 1 test in 7.237s
2022-12-01T11:08:36.4503756Z 
2022-12-01T11:08:36.4503840Z OK
2022-12-01T11:08:36.4503859Z 
2022-12-01T11:08:36.4503983Z Generating XML reports...
2022-12-01T11:08:36.4504436Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110742.xml
2022-12-01T11:08:36.4504809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4504994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4505365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4505559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4505579Z 
2022-12-01T11:08:36.4505685Z Running tests...
2022-12-01T11:08:36.4505952Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4506266Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4506547Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4506768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111172
2022-12-01T11:08:36.4506986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111173
2022-12-01T11:08:36.4507402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4507585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4507968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4508156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4508523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4508697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4509077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4509269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4510206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4510457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4510880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4511282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4511508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4511841Z STAGE:2022-12-01 11:07:55 111173:111173 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4512071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4512407Z STAGE:2022-12-01 11:07:55 111172:111172 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:08:36.4512752Z [1669892877.287156] [7ff5451e42d4:111172:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4512983Z [1669892877.287156] [7ff5451e42d4:111172:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4513313Z STAGE:2022-12-01 11:07:57 111172:111172 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4513536Z [1669892877.267059] [7ff5451e42d4:111173:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4513766Z [1669892877.267059] [7ff5451e42d4:111173:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4514099Z STAGE:2022-12-01 11:07:57 111173:111173 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:08:36.4514189Z ok (7.166s)
2022-12-01T11:08:36.4514216Z 
2022-12-01T11:08:36.4514474Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4514578Z Ran 1 test in 7.167s
2022-12-01T11:08:36.4514598Z 
2022-12-01T11:08:36.4514680Z OK
2022-12-01T11:08:36.4514699Z 
2022-12-01T11:08:36.4514814Z Generating XML reports...
2022-12-01T11:08:36.4515255Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110751.xml
2022-12-01T11:08:36.4515622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4515791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4516165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4516350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4516370Z 
2022-12-01T11:08:36.4516467Z Running tests...
2022-12-01T11:08:36.4516727Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4517098Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4517377Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T11:08:36.4517404Z 
2022-12-01T11:08:36.4517655Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4517756Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4517775Z 
2022-12-01T11:08:36.4517871Z OK (skipped=1)
2022-12-01T11:08:36.4517890Z 
2022-12-01T11:08:36.4518003Z Generating XML reports...
2022-12-01T11:08:36.4518440Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110801.xml
2022-12-01T11:08:36.4518802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4518973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4519346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4519519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4519546Z 
2022-12-01T11:08:36.4519637Z Running tests...
2022-12-01T11:08:36.4519889Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4520193Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4520470Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T11:08:36.4520491Z 
2022-12-01T11:08:36.4520741Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4520842Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4520862Z 
2022-12-01T11:08:36.4521018Z OK (skipped=1)
2022-12-01T11:08:36.4521038Z 
2022-12-01T11:08:36.4521149Z Generating XML reports...
2022-12-01T11:08:36.4521587Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110803.xml
2022-12-01T11:08:36.4521951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4522116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4522487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4522668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4522687Z 
2022-12-01T11:08:36.4522783Z Running tests...
2022-12-01T11:08:36.4523034Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4523338Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4523596Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4523813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111699
2022-12-01T11:08:36.4524024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111700
2022-12-01T11:08:36.4524394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4524559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4524930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4525110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4525469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4525637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4526048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4526239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4526470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4526712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4527117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4527516Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4527747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4527980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4528211Z [1669892890.646913] [7ff5451e42d4:111700:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4528436Z [1669892890.646913] [7ff5451e42d4:111700:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4528650Z [1669892890.638727] [7ff5451e42d4:111699:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4528870Z [1669892890.638727] [7ff5451e42d4:111699:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4528960Z ok (6.849s)
2022-12-01T11:08:36.4528980Z 
2022-12-01T11:08:36.4529237Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4529338Z Ran 1 test in 6.849s
2022-12-01T11:08:36.4529359Z 
2022-12-01T11:08:36.4529542Z OK
2022-12-01T11:08:36.4529562Z 
2022-12-01T11:08:36.4529677Z Generating XML reports...
2022-12-01T11:08:36.4530124Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110805.xml
2022-12-01T11:08:36.4530483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4530648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4531016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4531197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4531218Z 
2022-12-01T11:08:36.4531319Z Running tests...
2022-12-01T11:08:36.4531583Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4531897Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4532171Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4532377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112075
2022-12-01T11:08:36.4532598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112076
2022-12-01T11:08:36.4532971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4533149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4533531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4533723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4534095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4534276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4534702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4534884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4535130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4535373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4535782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4536179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4536410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4536674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17t_fahk
2022-12-01T11:08:36.4536950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17t_fahk/_remote_module_non_scriptable.py
2022-12-01T11:08:36.4537179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4537418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwomtrtk
2022-12-01T11:08:36.4537688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwomtrtk/_remote_module_non_scriptable.py
2022-12-01T11:08:36.4537922Z [1669892899.558344] [7ff5451e42d4:112075:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4538162Z [1669892899.558344] [7ff5451e42d4:112075:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4538388Z [1669892899.667353] [7ff5451e42d4:112076:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4538696Z [1669892899.667353] [7ff5451e42d4:112076:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4538799Z ok (6.186s)
2022-12-01T11:08:36.4538819Z 
2022-12-01T11:08:36.4539092Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4539204Z Ran 1 test in 6.186s
2022-12-01T11:08:36.4539224Z 
2022-12-01T11:08:36.4539297Z OK
2022-12-01T11:08:36.4539316Z 
2022-12-01T11:08:36.4539440Z Generating XML reports...
2022-12-01T11:08:36.4539890Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110814.xml
2022-12-01T11:08:36.4540264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4540441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4540826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4541024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4541043Z 
2022-12-01T11:08:36.4541156Z Running tests...
2022-12-01T11:08:36.4541406Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4541722Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4542027Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl & Gloo backend support DistributedDataParallel (0.002s)
2022-12-01T11:08:36.4542047Z 
2022-12-01T11:08:36.4542294Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4542394Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4542414Z 
2022-12-01T11:08:36.4542511Z OK (skipped=1)
2022-12-01T11:08:36.4542530Z 
2022-12-01T11:08:36.4542648Z Generating XML reports...
2022-12-01T11:08:36.4543141Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110822.xml
2022-12-01T11:08:36.4543518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4543677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4544046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4544227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4544247Z 
2022-12-01T11:08:36.4544342Z Running tests...
2022-12-01T11:08:36.4544596Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4544899Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4545180Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:08:36.4545398Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112529
2022-12-01T11:08:36.4545607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112530
2022-12-01T11:08:36.4545964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4546131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4546500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4546679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4547038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4547202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4547633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4547812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4548041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:08:36.4548276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:08:36.4548667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4549055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:08:36.4549274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:08:36.4550153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:08:36.4550432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgapor_rc
2022-12-01T11:08:36.4550694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgapor_rc/_remote_module_non_scriptable.py
2022-12-01T11:08:36.4550940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf70i4_8t
2022-12-01T11:08:36.4551189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf70i4_8t/_remote_module_non_scriptable.py
2022-12-01T11:08:36.4551412Z [1669892910.099523] [7ff5451e42d4:112530:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4551642Z [1669892910.099523] [7ff5451e42d4:112530:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4552506Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:08:36.4552750Z [1669892910.095506] [7ff5451e42d4:112529:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2022-12-01T11:08:36.4552975Z [1669892910.095506] [7ff5451e42d4:112529:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2022-12-01T11:08:36.4553743Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:08:36.4553839Z ok (6.468s)
2022-12-01T11:08:36.4553862Z 
2022-12-01T11:08:36.4554141Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4554244Z Ran 1 test in 6.468s
2022-12-01T11:08:36.4554265Z 
2022-12-01T11:08:36.4554346Z OK
2022-12-01T11:08:36.4554365Z 
2022-12-01T11:08:36.4554478Z Generating XML reports...
2022-12-01T11:08:36.4554923Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110824.xml
2022-12-01T11:08:36.4555283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4555450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4555823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4556083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4556104Z 
2022-12-01T11:08:36.4556200Z Running tests...
2022-12-01T11:08:36.4556456Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4556760Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4557184Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.4557205Z 
2022-12-01T11:08:36.4557458Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4557552Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4557571Z 
2022-12-01T11:08:36.4557670Z OK (skipped=1)
2022-12-01T11:08:36.4557689Z 
2022-12-01T11:08:36.4557801Z Generating XML reports...
2022-12-01T11:08:36.4558248Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110833.xml
2022-12-01T11:08:36.4558611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:08:36.4558776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:08:36.4559148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:08:36.4559329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:08:36.4559349Z 
2022-12-01T11:08:36.4559447Z Running tests...
2022-12-01T11:08:36.4559694Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4559995Z Test results will be stored in test-reports/dist-ucc/distributed.test_distributed_spawn
2022-12-01T11:08:36.4560426Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl', 'gloo'} (0.002s)
2022-12-01T11:08:36.4560501Z 
2022-12-01T11:08:36.4560764Z ----------------------------------------------------------------------
2022-12-01T11:08:36.4560865Z Ran 1 test in 0.002s
2022-12-01T11:08:36.4560885Z 
2022-12-01T11:08:36.4560980Z OK (skipped=1)
2022-12-01T11:08:36.4560999Z 
2022-12-01T11:08:36.4561113Z Generating XML reports...
2022-12-01T11:08:36.4561553Z Generated XML report: test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110835.xml
2022-12-01T11:08:36.4561573Z 
2022-12-01T11:08:36.4562227Z ##[endgroup]
2022-12-01T11:08:36.4562701Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_xjn_sjyx)
2022-12-01T11:08:36.4562721Z 
2022-12-01T11:12:12.6214694Z 
2022-12-01T11:12:12.6215906Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T11:12:12.6217340Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_0c81ynry)
2022-12-01T11:12:12.6223118Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:12:12.6280642Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T11:12:12.6345744Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6346716Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6347469Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6348249Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6349040Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6350385Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6351240Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6352074Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6353011Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6353958Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6354979Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6355900Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6356855Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6357761Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6358626Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6359506Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6360328Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6361230Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6361999Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6362804Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6363648Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6364519Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6365204Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6365904Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6366702Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6367442Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6368195Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6369175Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6369854Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6370228Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6370628Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6371033Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6371406Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6371810Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6372240Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6372638Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6373047Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6373468Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6373996Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6374402Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6374818Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6375261Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6375686Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6376138Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6376570Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6377000Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6377420Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6377855Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6378280Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6378707Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6379147Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6379561Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6379956Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6380385Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6380817Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6381222Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6381625Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6382040Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6382509Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6382894Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6383293Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6383694Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6384078Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6384435Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6384820Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6385228Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6385616Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6386014Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6386398Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6386765Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6387169Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6387566Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6387945Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6388363Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6388747Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6389122Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6390126Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6390541Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6390941Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6391325Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6391722Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6392201Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6392604Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6393041Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6393478Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6393921Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6394362Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6394816Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6395265Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6395693Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6396137Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6396582Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6397024Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6397456Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6397912Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6398374Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6398808Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6399260Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6399689Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6400082Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6400532Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6400903Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6401271Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6401634Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6402029Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6402420Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6402785Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6403188Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6403597Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6403997Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6404379Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6404786Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6405220Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6405620Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6406041Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6406456Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6406872Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6407285Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6407717Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6408130Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6408504Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6408874Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6409267Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6409692Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6410095Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6410494Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6410953Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6411448Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6411904Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6412320Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6412722Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6413155Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6413607Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6414085Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6414509Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6414935Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6415377Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6415771Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6416144Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6416538Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6416932Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6417348Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6417847Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6418287Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6418689Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6419101Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6419565Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6420048Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6420612Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6421224Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6421835Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6422425Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6423034Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6423636Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6424234Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6424817Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6425366Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6425917Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6426378Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6426748Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6427150Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6427563Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6427941Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6428366Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6428812Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6429275Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6430393Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6430821Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6431217Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6431617Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6432061Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6432491Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6432907Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6433306Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6433733Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6434164Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6434565Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6435077Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6435488Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6435910Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6436297Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6436716Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6437174Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6437582Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6437967Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6438367Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6438771Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6439191Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6439576Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6439941Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6440290Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6440670Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6441045Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6441396Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6441787Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6442169Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6442517Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6442874Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6443256Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6443657Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6444100Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6444491Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6444835Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6445202Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6445610Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6446022Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6446457Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6446912Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6447327Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6447747Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6448170Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6448596Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6449020Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6449417Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6449833Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6450246Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6450636Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6451040Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6451443Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6451907Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6452446Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6452917Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6453368Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6453809Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6454266Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6454697Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6455118Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6455532Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6455971Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6456423Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6456892Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6457405Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6457891Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6458309Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6458685Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6459087Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6459493Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6459864Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6460249Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6460642Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6461070Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6461451Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6461814Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6462190Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6462555Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6462957Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6463364Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6463722Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6464094Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6464481Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6464850Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6465216Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6465590Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6465971Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6466324Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6466710Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6467106Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6467472Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6467855Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6468227Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6468588Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6469065Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6470068Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6470614Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6470998Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6471402Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6471831Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6472229Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6472629Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6473048Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6473468Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6473885Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6474291Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6474701Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6475096Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6475482Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6475900Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6476334Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6476781Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6477517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6477975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6478541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6479012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6479253Z 
2022-12-01T11:12:12.6479364Z Running tests...
2022-12-01T11:12:12.6479832Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6480384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6480979Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6481541Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62019
2022-12-01T11:12:12.6481974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62020
2022-12-01T11:12:12.6482585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6483035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6483598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6484082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6484665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6485111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6485666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6486133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6486591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6487085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6487735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6488553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6489083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6489537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6490064Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6490898Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6491566Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6492372Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6493048Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6493872Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6494530Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6495353Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6495997Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6496880Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6497546Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6498362Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6499012Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6499832Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6500489Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:12:12.6501310Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:12:12.6501785Z ok (7.086s)
2022-12-01T11:12:12.6501936Z 
2022-12-01T11:12:12.6502211Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6502545Z Ran 1 test in 7.086s
2022-12-01T11:12:12.6502710Z 
2022-12-01T11:12:12.6502787Z OK
2022-12-01T11:12:12.6502923Z 
2022-12-01T11:12:12.6503052Z Generating XML reports...
2022-12-01T11:12:12.6503665Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104823.xml
2022-12-01T11:12:12.6504394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6504834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6505416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6505959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6506197Z 
2022-12-01T11:12:12.6506307Z Running tests...
2022-12-01T11:12:12.6506701Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6507236Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6507766Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-12-01T11:12:12.6508073Z 
2022-12-01T11:12:12.6508322Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6508653Z Ran 1 test in 0.004s
2022-12-01T11:12:12.6508815Z 
2022-12-01T11:12:12.6508927Z OK (skipped=1)
2022-12-01T11:12:12.6509081Z 
2022-12-01T11:12:12.6509205Z Generating XML reports...
2022-12-01T11:12:12.6510482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104832.xml
2022-12-01T11:12:12.6511218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6511672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6512234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6512711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6512944Z 
2022-12-01T11:12:12.6513053Z Running tests...
2022-12-01T11:12:12.6513461Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6513980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6514490Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6514986Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62460
2022-12-01T11:12:12.6515519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62461
2022-12-01T11:12:12.6516140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6516593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6517172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6517626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6518210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6518656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6519214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6519691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6520153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6520653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6521296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6521988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6522510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6523044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6523375Z ok (4.052s)
2022-12-01T11:12:12.6523528Z 
2022-12-01T11:12:12.6523798Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6524211Z Ran 1 test in 4.052s
2022-12-01T11:12:12.6524376Z 
2022-12-01T11:12:12.6524451Z OK
2022-12-01T11:12:12.6524588Z 
2022-12-01T11:12:12.6524716Z Generating XML reports...
2022-12-01T11:12:12.6525331Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104834.xml
2022-12-01T11:12:12.6526049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6526483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6527062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6527533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6527763Z 
2022-12-01T11:12:12.6527854Z Running tests...
2022-12-01T11:12:12.6528265Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6528806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6529337Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6530383Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.707s)
2022-12-01T11:12:12.6530918Z 
2022-12-01T11:12:12.6531186Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6531515Z Ran 1 test in 1.707s
2022-12-01T11:12:12.6531678Z 
2022-12-01T11:12:12.6531789Z OK (skipped=1)
2022-12-01T11:12:12.6531943Z 
2022-12-01T11:12:12.6532053Z Generating XML reports...
2022-12-01T11:12:12.6532667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104841.xml
2022-12-01T11:12:12.6533489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6533955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6534522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6534998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6535227Z 
2022-12-01T11:12:12.6535337Z Running tests...
2022-12-01T11:12:12.6535726Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6536261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6536816Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-12-01T11:12:12.6537147Z 
2022-12-01T11:12:12.6537420Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6537730Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6537888Z 
2022-12-01T11:12:12.6537996Z OK (skipped=1)
2022-12-01T11:12:12.6538148Z 
2022-12-01T11:12:12.6538271Z Generating XML reports...
2022-12-01T11:12:12.6538858Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104845.xml
2022-12-01T11:12:12.6539572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6540018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6540591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6541046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6541341Z 
2022-12-01T11:12:12.6541451Z Running tests...
2022-12-01T11:12:12.6541861Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6542377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6542944Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-12-01T11:12:12.6543285Z 
2022-12-01T11:12:12.6543550Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6543871Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6544031Z 
2022-12-01T11:12:12.6544121Z OK (skipped=1)
2022-12-01T11:12:12.6544274Z 
2022-12-01T11:12:12.6544398Z Generating XML reports...
2022-12-01T11:12:12.6545001Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104847.xml
2022-12-01T11:12:12.6545717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6546158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6546736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6547202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6547429Z 
2022-12-01T11:12:12.6547522Z Running tests...
2022-12-01T11:12:12.6547929Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6548459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6549011Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6550193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63136
2022-12-01T11:12:12.6550667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63137
2022-12-01T11:12:12.6551394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6551844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6552429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6552901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6553480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6553907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6554483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6554949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6555393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6555893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6556555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6557246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6557750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6558229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6558724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwr7b2ud
2022-12-01T11:12:12.6559267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwr7b2ud/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6559874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc140pe4a
2022-12-01T11:12:12.6560408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc140pe4a/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6561320Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6561866Z   warnings.warn(
2022-12-01T11:12:12.6562605Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6563139Z   warnings.warn(
2022-12-01T11:12:12.6563515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6563991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6564474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6564956Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6565432Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6565891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6566364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6566836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6567303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6567758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6568281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6568761Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6569141Z ok (7.777s)
2022-12-01T11:12:12.6569290Z 
2022-12-01T11:12:12.6569567Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6569898Z Ran 1 test in 7.778s
2022-12-01T11:12:12.6570058Z 
2022-12-01T11:12:12.6570149Z OK
2022-12-01T11:12:12.6570264Z 
2022-12-01T11:12:12.6570388Z Generating XML reports...
2022-12-01T11:12:12.6571005Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104849.xml
2022-12-01T11:12:12.6571713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6572155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6572740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6573214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6573443Z 
2022-12-01T11:12:12.6573554Z Running tests...
2022-12-01T11:12:12.6573944Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6574477Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6575039Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6575564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63583
2022-12-01T11:12:12.6576010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63584
2022-12-01T11:12:12.6576621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6577155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6577722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6578195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6578783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6579209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6579789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6580258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6580712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6581204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6581863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6582555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6583079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6583536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6584032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezbbvhrp
2022-12-01T11:12:12.6584580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezbbvhrp/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6585093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_xz58rx7
2022-12-01T11:12:12.6585685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_xz58rx7/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6586606Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6587156Z   warnings.warn(
2022-12-01T11:12:12.6587893Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6588432Z   warnings.warn(
2022-12-01T11:12:12.6588807Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6589298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6590361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6590854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6591209Z ok (7.012s)
2022-12-01T11:12:12.6591358Z 
2022-12-01T11:12:12.6591621Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6591954Z Ran 1 test in 7.013s
2022-12-01T11:12:12.6592114Z 
2022-12-01T11:12:12.6592210Z OK
2022-12-01T11:12:12.6592343Z 
2022-12-01T11:12:12.6592467Z Generating XML reports...
2022-12-01T11:12:12.6593068Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104859.xml
2022-12-01T11:12:12.6593794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6594250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6594815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6595391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6595623Z 
2022-12-01T11:12:12.6595730Z Running tests...
2022-12-01T11:12:12.6596147Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6596664Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6597238Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6597783Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63935
2022-12-01T11:12:12.6598231Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63936
2022-12-01T11:12:12.6598823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6599284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6599860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6600313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6600897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6601341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6601911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6602360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6602814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6603313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6604030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6604742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6605264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6605735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6606222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkq43ntt
2022-12-01T11:12:12.6606763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkq43ntt/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6607293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fag2qnk
2022-12-01T11:12:12.6607836Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fag2qnk/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6608735Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6609284Z   warnings.warn(
2022-12-01T11:12:12.6610041Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6610583Z   warnings.warn(
2022-12-01T11:12:12.6610945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6611433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6611913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6612443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6612801Z ok (7.028s)
2022-12-01T11:12:12.6612950Z 
2022-12-01T11:12:12.6613229Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6613563Z Ran 1 test in 7.028s
2022-12-01T11:12:12.6613706Z 
2022-12-01T11:12:12.6613805Z OK
2022-12-01T11:12:12.6613939Z 
2022-12-01T11:12:12.6614063Z Generating XML reports...
2022-12-01T11:12:12.6614678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104908.xml
2022-12-01T11:12:12.6615382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6615840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6616421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6616898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6617113Z 
2022-12-01T11:12:12.6617226Z Running tests...
2022-12-01T11:12:12.6617633Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6618173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6618746Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6619314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64298
2022-12-01T11:12:12.6619765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64299
2022-12-01T11:12:12.6620378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6620816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6621458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6621939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6622523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6622958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6623535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6624002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6624441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6624945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6625617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6626309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6626816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6627290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6627788Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3c2jqlwk
2022-12-01T11:12:12.6628329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3c2jqlwk/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6628848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe2a87ahq
2022-12-01T11:12:12.6629720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe2a87ahq/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6630889Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6631444Z   warnings.warn(
2022-12-01T11:12:12.6632182Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6632716Z   warnings.warn(
2022-12-01T11:12:12.6633099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6633576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6633928Z ok (7.116s)
2022-12-01T11:12:12.6634077Z 
2022-12-01T11:12:12.6634361Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6634697Z Ran 1 test in 7.116s
2022-12-01T11:12:12.6634839Z 
2022-12-01T11:12:12.6634933Z OK
2022-12-01T11:12:12.6635068Z 
2022-12-01T11:12:12.6635193Z Generating XML reports...
2022-12-01T11:12:12.6635806Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104917.xml
2022-12-01T11:12:12.6636505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6636957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6637536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6638015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6638226Z 
2022-12-01T11:12:12.6638336Z Running tests...
2022-12-01T11:12:12.6638750Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6639361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6639947Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6640505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64655
2022-12-01T11:12:12.6640951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64656
2022-12-01T11:12:12.6641565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6641998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6642577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6643051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6643622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6644068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6644639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6645105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6645540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6646040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6646691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6647387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6647958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6648431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6648928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hi_nhs2
2022-12-01T11:12:12.6649444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hi_nhs2/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6649973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2ouq6x8
2022-12-01T11:12:12.6650504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2ouq6x8/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6651414Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6651948Z   warnings.warn(
2022-12-01T11:12:12.6652710Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6653260Z   warnings.warn(
2022-12-01T11:12:12.6653637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6654112Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6654592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6655071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6655419Z ok (8.017s)
2022-12-01T11:12:12.6655551Z 
2022-12-01T11:12:12.6655825Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6656162Z Ran 1 test in 8.017s
2022-12-01T11:12:12.6656373Z 
2022-12-01T11:12:12.6656472Z OK
2022-12-01T11:12:12.6656590Z 
2022-12-01T11:12:12.6656718Z Generating XML reports...
2022-12-01T11:12:12.6657334Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104927.xml
2022-12-01T11:12:12.6658055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6658507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6659065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6659542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6659771Z 
2022-12-01T11:12:12.6659880Z Running tests...
2022-12-01T11:12:12.6660267Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6660807Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6661372Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6661915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65042
2022-12-01T11:12:12.6662347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65043
2022-12-01T11:12:12.6662955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6663409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6663969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6664444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6665088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6665537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6666098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6666567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6667025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6667504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6668165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6668855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6669489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6670325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6670836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ylm6nd5
2022-12-01T11:12:12.6671376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ylm6nd5/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6671913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw5oi_g8
2022-12-01T11:12:12.6672428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw5oi_g8/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6673345Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6673899Z   warnings.warn(
2022-12-01T11:12:12.6674745Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6675284Z   warnings.warn(
2022-12-01T11:12:12.6675665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6676152Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6676614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6677097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6677450Z ok (7.420s)
2022-12-01T11:12:12.6677599Z 
2022-12-01T11:12:12.6677877Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6678197Z Ran 1 test in 7.420s
2022-12-01T11:12:12.6678357Z 
2022-12-01T11:12:12.6678453Z OK
2022-12-01T11:12:12.6678589Z 
2022-12-01T11:12:12.6678714Z Generating XML reports...
2022-12-01T11:12:12.6679310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104937.xml
2022-12-01T11:12:12.6680028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6680474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6681054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6681512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6681742Z 
2022-12-01T11:12:12.6681850Z Running tests...
2022-12-01T11:12:12.6682258Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6682862Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6683457Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6684020Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65457
2022-12-01T11:12:12.6684470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65458
2022-12-01T11:12:12.6685066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6685520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6686096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6686568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6687136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6687588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6688165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6688615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6689067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6689561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6690219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6690893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6691475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6691957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6692455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fzgdd0y
2022-12-01T11:12:12.6692979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fzgdd0y/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6693513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp048enn2a
2022-12-01T11:12:12.6694041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp048enn2a/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6694934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6695488Z   warnings.warn(
2022-12-01T11:12:12.6696245Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.6696788Z   warnings.warn(
2022-12-01T11:12:12.6697147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6697637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6698116Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6698594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6698922Z ok (6.879s)
2022-12-01T11:12:12.6699070Z 
2022-12-01T11:12:12.6699346Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6699676Z Ran 1 test in 6.879s
2022-12-01T11:12:12.6699900Z 
2022-12-01T11:12:12.6699978Z OK
2022-12-01T11:12:12.6700116Z 
2022-12-01T11:12:12.6700244Z Generating XML reports...
2022-12-01T11:12:12.6700858Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104947.xml
2022-12-01T11:12:12.6701577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6702011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6702589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6703064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6703294Z 
2022-12-01T11:12:12.6703388Z Running tests...
2022-12-01T11:12:12.6703793Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6704328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6704894Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6705954Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.618s)
2022-12-01T11:12:12.6706474Z 
2022-12-01T11:12:12.6706742Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6707071Z Ran 1 test in 1.618s
2022-12-01T11:12:12.6707233Z 
2022-12-01T11:12:12.6707341Z OK (skipped=1)
2022-12-01T11:12:12.6707496Z 
2022-12-01T11:12:12.6707604Z Generating XML reports...
2022-12-01T11:12:12.6708212Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104956.xml
2022-12-01T11:12:12.6708979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6709708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6710302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6710774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6711004Z 
2022-12-01T11:12:12.6711114Z Running tests...
2022-12-01T11:12:12.6711500Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6712038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6712583Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6713109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65898
2022-12-01T11:12:12.6713553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65899
2022-12-01T11:12:12.6714159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6714608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6715167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6715636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6716218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6716662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6717220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6717790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6718244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6718743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6719393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6720089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6720611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6721067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6721412Z ok (4.150s)
2022-12-01T11:12:12.6721562Z 
2022-12-01T11:12:12.6721839Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6722172Z Ran 1 test in 4.150s
2022-12-01T11:12:12.6722314Z 
2022-12-01T11:12:12.6722406Z OK
2022-12-01T11:12:12.6722542Z 
2022-12-01T11:12:12.6722667Z Generating XML reports...
2022-12-01T11:12:12.6723276Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105000.xml
2022-12-01T11:12:12.6723977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6724429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6725006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6725478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6725689Z 
2022-12-01T11:12:12.6725799Z Running tests...
2022-12-01T11:12:12.6726211Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6726816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6727367Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6728453Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.656s)
2022-12-01T11:12:12.6728975Z 
2022-12-01T11:12:12.6729241Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6729568Z Ran 1 test in 1.657s
2022-12-01T11:12:12.6729728Z 
2022-12-01T11:12:12.6729835Z OK (skipped=1)
2022-12-01T11:12:12.6729971Z 
2022-12-01T11:12:12.6730095Z Generating XML reports...
2022-12-01T11:12:12.6730716Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105006.xml
2022-12-01T11:12:12.6731431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6731861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6732435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6732906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6733132Z 
2022-12-01T11:12:12.6733242Z Running tests...
2022-12-01T11:12:12.6733630Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6734164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6734692Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6735243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66317
2022-12-01T11:12:12.6735690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66318
2022-12-01T11:12:12.6736300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6736750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6737309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6737776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6738361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6738788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6739371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6739836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6740291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6740767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6741420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6742109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6742629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6743084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6743429Z ok (6.974s)
2022-12-01T11:12:12.6743576Z 
2022-12-01T11:12:12.6743917Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6744235Z Ran 1 test in 6.974s
2022-12-01T11:12:12.6744396Z 
2022-12-01T11:12:12.6744489Z OK
2022-12-01T11:12:12.6744622Z 
2022-12-01T11:12:12.6744746Z Generating XML reports...
2022-12-01T11:12:12.6745360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105010.xml
2022-12-01T11:12:12.6746059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6746505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6747080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6747535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6747773Z 
2022-12-01T11:12:12.6747880Z Running tests...
2022-12-01T11:12:12.6748286Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6748820Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6749303Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T11:12:12.6750048Z 
2022-12-01T11:12:12.6750331Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6750657Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6750819Z 
2022-12-01T11:12:12.6750910Z OK (skipped=1)
2022-12-01T11:12:12.6751064Z 
2022-12-01T11:12:12.6751190Z Generating XML reports...
2022-12-01T11:12:12.6751800Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105019.xml
2022-12-01T11:12:12.6752513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6753065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6753647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6754117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6754347Z 
2022-12-01T11:12:12.6754461Z Running tests...
2022-12-01T11:12:12.6754849Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6755378Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6755838Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6756300Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6756781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66821
2022-12-01T11:12:12.6757234Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66822
2022-12-01T11:12:12.6757827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6758284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6758861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6759329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6759885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6760329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6760902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6761367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6761877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6762385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6763052Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6763721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6764243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6764717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6765221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5xy3t1e
2022-12-01T11:12:12.6765745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5xy3t1e/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6766286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnjubk6a2
2022-12-01T11:12:12.6766823Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnjubk6a2/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6767206Z ok (7.329s)
2022-12-01T11:12:12.6767336Z 
2022-12-01T11:12:12.6767611Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6767941Z Ran 1 test in 7.330s
2022-12-01T11:12:12.6768103Z 
2022-12-01T11:12:12.6768197Z OK
2022-12-01T11:12:12.6768331Z 
2022-12-01T11:12:12.6768436Z Generating XML reports...
2022-12-01T11:12:12.6769095Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105021.xml
2022-12-01T11:12:12.6769822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6770338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6770907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6771373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6771601Z 
2022-12-01T11:12:12.6771709Z Running tests...
2022-12-01T11:12:12.6772097Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6772632Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6773111Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6773613Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6774081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67233
2022-12-01T11:12:12.6774532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67234
2022-12-01T11:12:12.6775151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6775579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6776159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6776629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6777210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6777638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6778208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6778672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6779169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6779676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6780337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6781025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6781528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6781998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6782490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__e9xilz
2022-12-01T11:12:12.6783027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__e9xilz/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6783545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr6w34hy
2022-12-01T11:12:12.6784081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr6w34hy/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6784597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6785063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6785415Z ok (7.287s)
2022-12-01T11:12:12.6785564Z 
2022-12-01T11:12:12.6785838Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6786165Z Ran 1 test in 7.288s
2022-12-01T11:12:12.6786307Z 
2022-12-01T11:12:12.6786401Z OK
2022-12-01T11:12:12.6786533Z 
2022-12-01T11:12:12.6786655Z Generating XML reports...
2022-12-01T11:12:12.6787264Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105031.xml
2022-12-01T11:12:12.6788039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6788495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6789081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6789775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6790013Z 
2022-12-01T11:12:12.6790122Z Running tests...
2022-12-01T11:12:12.6790534Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6791067Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6791547Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6792080Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6792590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67681
2022-12-01T11:12:12.6793023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67682
2022-12-01T11:12:12.6793635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6794084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6794656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6795109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6795688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6796128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6796765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6797244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6797699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6798198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6798843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6799531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6800050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6800520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6801012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgnbtcwe9
2022-12-01T11:12:12.6801557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgnbtcwe9/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6802090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeshcfhs_
2022-12-01T11:12:12.6802632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeshcfhs_/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6803125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6803614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.6803968Z ok (7.124s)
2022-12-01T11:12:12.6804115Z 
2022-12-01T11:12:12.6804371Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6804700Z Ran 1 test in 7.124s
2022-12-01T11:12:12.6804961Z 
2022-12-01T11:12:12.6805054Z OK
2022-12-01T11:12:12.6805186Z 
2022-12-01T11:12:12.6805292Z Generating XML reports...
2022-12-01T11:12:12.6805911Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105040.xml
2022-12-01T11:12:12.6806627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6807079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6807641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6808105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6808334Z 
2022-12-01T11:12:12.6808444Z Running tests...
2022-12-01T11:12:12.6808852Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6809363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6809844Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:12:12.6810340Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6810804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68090
2022-12-01T11:12:12.6811250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68091
2022-12-01T11:12:12.6811856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6812298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6812858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6813323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6813898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6814379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6814962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6815421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6815875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6816350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6817002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6817691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6818219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6818678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6819176Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhvrtfs5
2022-12-01T11:12:12.6819714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhvrtfs5/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6820228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2a3pkcw0
2022-12-01T11:12:12.6820761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2a3pkcw0/_remote_module_non_scriptable.py
2022-12-01T11:12:12.6821147Z ok (7.289s)
2022-12-01T11:12:12.6821296Z 
2022-12-01T11:12:12.6821567Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6821877Z Ran 1 test in 7.289s
2022-12-01T11:12:12.6822037Z 
2022-12-01T11:12:12.6822129Z OK
2022-12-01T11:12:12.6822324Z 
2022-12-01T11:12:12.6822449Z Generating XML reports...
2022-12-01T11:12:12.6823106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105050.xml
2022-12-01T11:12:12.6823825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6824276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6824854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6825308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6825540Z 
2022-12-01T11:12:12.6825650Z Running tests...
2022-12-01T11:12:12.6826054Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6826570Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6827083Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.6827369Z 
2022-12-01T11:12:12.6827635Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6827962Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6828103Z 
2022-12-01T11:12:12.6828209Z OK (skipped=1)
2022-12-01T11:12:12.6828363Z 
2022-12-01T11:12:12.6828487Z Generating XML reports...
2022-12-01T11:12:12.6829097Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105059.xml
2022-12-01T11:12:12.6830038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6830489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6831063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6831533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6831864Z 
2022-12-01T11:12:12.6831966Z Running tests...
2022-12-01T11:12:12.6832377Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6832906Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6833443Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T11:12:12.6833744Z 
2022-12-01T11:12:12.6834014Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6834342Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6834501Z 
2022-12-01T11:12:12.6834608Z OK (skipped=1)
2022-12-01T11:12:12.6834760Z 
2022-12-01T11:12:12.6834865Z Generating XML reports...
2022-12-01T11:12:12.6835472Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105101.xml
2022-12-01T11:12:12.6836193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6836645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6837201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6837669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6837899Z 
2022-12-01T11:12:12.6838007Z Running tests...
2022-12-01T11:12:12.6838392Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6838924Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6839461Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T11:12:12.6839785Z 
2022-12-01T11:12:12.6840137Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6840453Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6840614Z 
2022-12-01T11:12:12.6840725Z OK (skipped=1)
2022-12-01T11:12:12.6840879Z 
2022-12-01T11:12:12.6841003Z Generating XML reports...
2022-12-01T11:12:12.6841593Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105104.xml
2022-12-01T11:12:12.6842311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6842763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6843342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6843794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6844022Z 
2022-12-01T11:12:12.6844129Z Running tests...
2022-12-01T11:12:12.6844537Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6845070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6845592Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T11:12:12.6845906Z 
2022-12-01T11:12:12.6846172Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6846493Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6846653Z 
2022-12-01T11:12:12.6846744Z OK (skipped=1)
2022-12-01T11:12:12.6846900Z 
2022-12-01T11:12:12.6847021Z Generating XML reports...
2022-12-01T11:12:12.6847625Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105106.xml
2022-12-01T11:12:12.6848343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6848778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6849406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6849888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6850116Z 
2022-12-01T11:12:12.6850206Z Running tests...
2022-12-01T11:12:12.6850613Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6851147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6851687Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2022-12-01T11:12:12.6852007Z 
2022-12-01T11:12:12.6852257Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6852583Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6852742Z 
2022-12-01T11:12:12.6852851Z OK (skipped=1)
2022-12-01T11:12:12.6853011Z 
2022-12-01T11:12:12.6853136Z Generating XML reports...
2022-12-01T11:12:12.6853726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105108.xml
2022-12-01T11:12:12.6854434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6854886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6855443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6855636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6855655Z 
2022-12-01T11:12:12.6855766Z Running tests...
2022-12-01T11:12:12.6856035Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6856351Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6856709Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.003s)
2022-12-01T11:12:12.6856729Z 
2022-12-01T11:12:12.6856994Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6857107Z Ran 1 test in 0.003s
2022-12-01T11:12:12.6857127Z 
2022-12-01T11:12:12.6857216Z OK (skipped=1)
2022-12-01T11:12:12.6857254Z 
2022-12-01T11:12:12.6857360Z Generating XML reports...
2022-12-01T11:12:12.6857809Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105110.xml
2022-12-01T11:12:12.6858182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6858359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6858737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6858934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6858954Z 
2022-12-01T11:12:12.6859065Z Running tests...
2022-12-01T11:12:12.6859330Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6859624Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6859895Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.6859915Z 
2022-12-01T11:12:12.6860178Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6860289Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6860309Z 
2022-12-01T11:12:12.6860418Z OK (skipped=1)
2022-12-01T11:12:12.6860437Z 
2022-12-01T11:12:12.6860562Z Generating XML reports...
2022-12-01T11:12:12.6861008Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105112.xml
2022-12-01T11:12:12.6861448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6861631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6861999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6862192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6862212Z 
2022-12-01T11:12:12.6862323Z Running tests...
2022-12-01T11:12:12.6862583Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6862897Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6863153Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6863377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69224
2022-12-01T11:12:12.6863599Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69225
2022-12-01T11:12:12.6863956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6864133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6864503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6864677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6865058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6865252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6865624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6865863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6866114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6866334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6866740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6867143Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6867375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6867601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6867938Z STAGE:2022-12-01 10:51:19 69225:69225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6868271Z STAGE:2022-12-01 10:51:19 69224:69224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6868611Z STAGE:2022-12-01 10:51:21 69224:69224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6868945Z STAGE:2022-12-01 10:51:21 69225:69225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6869306Z STAGE:2022-12-01 10:51:21 69225:69225 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6869840Z STAGE:2022-12-01 10:51:21 69224:69224 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6870190Z STAGE:2022-12-01 10:51:21 69225:69225 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6870523Z STAGE:2022-12-01 10:51:21 69224:69224 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6870628Z ok (7.694s)
2022-12-01T11:12:12.6870649Z 
2022-12-01T11:12:12.6870920Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6871106Z Ran 1 test in 7.694s
2022-12-01T11:12:12.6871128Z 
2022-12-01T11:12:12.6871225Z OK
2022-12-01T11:12:12.6871244Z 
2022-12-01T11:12:12.6871352Z Generating XML reports...
2022-12-01T11:12:12.6871811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105114.xml
2022-12-01T11:12:12.6872185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6872362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6872745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6872939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6872959Z 
2022-12-01T11:12:12.6873068Z Running tests...
2022-12-01T11:12:12.6873339Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6873661Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6873914Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6874136Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69598
2022-12-01T11:12:12.6874348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69599
2022-12-01T11:12:12.6874723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6874900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6875284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6875479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6875930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6876087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6876465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6876655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6876900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6877138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6877541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6877940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6878179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6878411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6878729Z STAGE:2022-12-01 10:51:29 69599:69599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6879067Z STAGE:2022-12-01 10:51:29 69598:69598 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6879405Z STAGE:2022-12-01 10:51:31 69598:69598 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6879742Z STAGE:2022-12-01 10:51:31 69599:69599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6880065Z STAGE:2022-12-01 10:51:31 69598:69598 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6880401Z STAGE:2022-12-01 10:51:31 69598:69598 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6880779Z STAGE:2022-12-01 10:51:31 69599:69599 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6881121Z STAGE:2022-12-01 10:51:31 69599:69599 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6881223Z ok (7.710s)
2022-12-01T11:12:12.6881243Z 
2022-12-01T11:12:12.6881492Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6881606Z Ran 1 test in 7.710s
2022-12-01T11:12:12.6881625Z 
2022-12-01T11:12:12.6881717Z OK
2022-12-01T11:12:12.6881736Z 
2022-12-01T11:12:12.6881858Z Generating XML reports...
2022-12-01T11:12:12.6882313Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105124.xml
2022-12-01T11:12:12.6882687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6882865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6883256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6883431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6883470Z 
2022-12-01T11:12:12.6883559Z Running tests...
2022-12-01T11:12:12.6883823Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6884137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6884405Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.6884425Z 
2022-12-01T11:12:12.6884688Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6884801Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6884821Z 
2022-12-01T11:12:12.6884935Z OK (skipped=1)
2022-12-01T11:12:12.6884954Z 
2022-12-01T11:12:12.6885133Z Generating XML reports...
2022-12-01T11:12:12.6885574Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105134.xml
2022-12-01T11:12:12.6885949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6886123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6886508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6886701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6886721Z 
2022-12-01T11:12:12.6886829Z Running tests...
2022-12-01T11:12:12.6887090Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6887411Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6887673Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.6887699Z 
2022-12-01T11:12:12.6887944Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6888056Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6888075Z 
2022-12-01T11:12:12.6888180Z OK (skipped=1)
2022-12-01T11:12:12.6888199Z 
2022-12-01T11:12:12.6888321Z Generating XML reports...
2022-12-01T11:12:12.6888769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105136.xml
2022-12-01T11:12:12.6889140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6889318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6889704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6889897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6889920Z 
2022-12-01T11:12:12.6890060Z Running tests...
2022-12-01T11:12:12.6890332Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6890645Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6890925Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6891147Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70309
2022-12-01T11:12:12.6891362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70310
2022-12-01T11:12:12.6891730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6891907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6892269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6892468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6892830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6893007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6893385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6893578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6893822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6894060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6894466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6894911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6895147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6895376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6895716Z STAGE:2022-12-01 10:51:43 70310:70310 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6896048Z STAGE:2022-12-01 10:51:43 70309:70309 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6896383Z STAGE:2022-12-01 10:51:44 70309:70309 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6896720Z STAGE:2022-12-01 10:51:44 70310:70310 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6896823Z ok (7.735s)
2022-12-01T11:12:12.6896847Z 
2022-12-01T11:12:12.6897116Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6897215Z Ran 1 test in 7.736s
2022-12-01T11:12:12.6897233Z 
2022-12-01T11:12:12.6897324Z OK
2022-12-01T11:12:12.6897343Z 
2022-12-01T11:12:12.6897468Z Generating XML reports...
2022-12-01T11:12:12.6897921Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105138.xml
2022-12-01T11:12:12.6898299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6898476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6898857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6899050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6899070Z 
2022-12-01T11:12:12.6899162Z Running tests...
2022-12-01T11:12:12.6899432Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6899792Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6900080Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6900301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70767
2022-12-01T11:12:12.6900513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70768
2022-12-01T11:12:12.6900887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6901062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6901426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6901620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6901993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6902168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6902550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6902742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6902991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6903230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6903630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6904010Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6904298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6904528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6904866Z STAGE:2022-12-01 10:51:53 70768:70768 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6905199Z STAGE:2022-12-01 10:51:53 70767:70767 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.6905534Z STAGE:2022-12-01 10:51:54 70767:70767 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6905869Z STAGE:2022-12-01 10:51:54 70768:70768 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.6905973Z ok (7.589s)
2022-12-01T11:12:12.6905994Z 
2022-12-01T11:12:12.6906259Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6906361Z Ran 1 test in 7.590s
2022-12-01T11:12:12.6906380Z 
2022-12-01T11:12:12.6906473Z OK
2022-12-01T11:12:12.6906495Z 
2022-12-01T11:12:12.6906620Z Generating XML reports...
2022-12-01T11:12:12.6907076Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105148.xml
2022-12-01T11:12:12.6907449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6907626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6908006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6908200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6908220Z 
2022-12-01T11:12:12.6908328Z Running tests...
2022-12-01T11:12:12.6908577Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6908896Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6909207Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6910231Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2022-12-01T11:12:12.6910254Z 
2022-12-01T11:12:12.6910518Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6910630Z Ran 1 test in 1.666s
2022-12-01T11:12:12.6910650Z 
2022-12-01T11:12:12.6910758Z OK (skipped=1)
2022-12-01T11:12:12.6910777Z 
2022-12-01T11:12:12.6910902Z Generating XML reports...
2022-12-01T11:12:12.6911352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105158.xml
2022-12-01T11:12:12.6911719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6911898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6912278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6912470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6912490Z 
2022-12-01T11:12:12.6912598Z Running tests...
2022-12-01T11:12:12.6912861Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6913178Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6913453Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6914202Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.749s)
2022-12-01T11:12:12.6914296Z 
2022-12-01T11:12:12.6914570Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6914668Z Ran 1 test in 1.749s
2022-12-01T11:12:12.6914688Z 
2022-12-01T11:12:12.6914798Z OK (skipped=1)
2022-12-01T11:12:12.6914817Z 
2022-12-01T11:12:12.6914943Z Generating XML reports...
2022-12-01T11:12:12.6915394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105202.xml
2022-12-01T11:12:12.6915764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6915939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6916330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6916524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6916543Z 
2022-12-01T11:12:12.6916634Z Running tests...
2022-12-01T11:12:12.6916899Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6917211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6917487Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6917708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71534
2022-12-01T11:12:12.6917918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71535
2022-12-01T11:12:12.6918296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6918480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6918924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6919107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6919482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6919660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6920041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6920233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6920480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6920720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6921130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6921531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6921746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6921975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6922078Z ok (6.997s)
2022-12-01T11:12:12.6922098Z 
2022-12-01T11:12:12.6922365Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6922478Z Ran 1 test in 6.997s
2022-12-01T11:12:12.6922498Z 
2022-12-01T11:12:12.6922591Z OK
2022-12-01T11:12:12.6922610Z 
2022-12-01T11:12:12.6922738Z Generating XML reports...
2022-12-01T11:12:12.6923255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105206.xml
2022-12-01T11:12:12.6923613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6923789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6924169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6924364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6924385Z 
2022-12-01T11:12:12.6924498Z Running tests...
2022-12-01T11:12:12.6924760Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6925072Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6925345Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6925572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71969
2022-12-01T11:12:12.6925768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71970
2022-12-01T11:12:12.6926141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6926316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6926700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6926894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6927263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6927440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6927822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6928057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6928313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6928552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6928956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6929353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6929584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6929811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6930061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.6930300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.6930683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.6931076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.6931316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.6931561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.6931954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.6932346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.6932648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-12-01T11:12:12.6932886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-12-01T11:12:12.6933282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T11:12:12.6933657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T11:12:12.6964205Z ok (8.493s)
2022-12-01T11:12:12.6964238Z 
2022-12-01T11:12:12.6964610Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6964718Z Ran 1 test in 8.493s
2022-12-01T11:12:12.6964738Z 
2022-12-01T11:12:12.6964833Z OK
2022-12-01T11:12:12.6964852Z 
2022-12-01T11:12:12.6964984Z Generating XML reports...
2022-12-01T11:12:12.6965477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105215.xml
2022-12-01T11:12:12.6965860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6966043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6966431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6966628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6966648Z 
2022-12-01T11:12:12.6966761Z Running tests...
2022-12-01T11:12:12.6967009Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6967332Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6967595Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.6967966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72474
2022-12-01T11:12:12.6968207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72475
2022-12-01T11:12:12.6968587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6968766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6969202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6969379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6969748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6969925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6970315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6970512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6970760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.6971007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.6971415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6971819Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.6972035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.6972265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.6972447Z ok (7.084s)
2022-12-01T11:12:12.6972467Z 
2022-12-01T11:12:12.6972741Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6972856Z Ran 1 test in 7.084s
2022-12-01T11:12:12.6972876Z 
2022-12-01T11:12:12.6972969Z OK
2022-12-01T11:12:12.6972987Z 
2022-12-01T11:12:12.6973115Z Generating XML reports...
2022-12-01T11:12:12.6973572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105226.xml
2022-12-01T11:12:12.6973931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6974111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6974497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6974694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6974717Z 
2022-12-01T11:12:12.6974829Z Running tests...
2022-12-01T11:12:12.6975101Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6975419Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6975847Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6975867Z 
2022-12-01T11:12:12.6976135Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6976233Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6976252Z 
2022-12-01T11:12:12.6976365Z OK (skipped=1)
2022-12-01T11:12:12.6976384Z 
2022-12-01T11:12:12.6976512Z Generating XML reports...
2022-12-01T11:12:12.6976965Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105235.xml
2022-12-01T11:12:12.6977342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6977575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6977973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6978165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6978184Z 
2022-12-01T11:12:12.6978284Z Running tests...
2022-12-01T11:12:12.6978532Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6978847Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6979258Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6979278Z 
2022-12-01T11:12:12.6979538Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6979653Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6979672Z 
2022-12-01T11:12:12.6979776Z OK (skipped=1)
2022-12-01T11:12:12.6979799Z 
2022-12-01T11:12:12.6979917Z Generating XML reports...
2022-12-01T11:12:12.6980363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105237.xml
2022-12-01T11:12:12.6980727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6980891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6981265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6981448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6981467Z 
2022-12-01T11:12:12.6981569Z Running tests...
2022-12-01T11:12:12.6981834Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6982208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6982644Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6982664Z 
2022-12-01T11:12:12.6982925Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6983019Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6983058Z 
2022-12-01T11:12:12.6983150Z OK (skipped=1)
2022-12-01T11:12:12.6983169Z 
2022-12-01T11:12:12.6983297Z Generating XML reports...
2022-12-01T11:12:12.6983752Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105239.xml
2022-12-01T11:12:12.6984129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6984303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6984696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6984893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6984912Z 
2022-12-01T11:12:12.6985022Z Running tests...
2022-12-01T11:12:12.6985270Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6985587Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6986007Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6986027Z 
2022-12-01T11:12:12.6986288Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6986402Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6986421Z 
2022-12-01T11:12:12.6986530Z OK (skipped=1)
2022-12-01T11:12:12.6986548Z 
2022-12-01T11:12:12.6986678Z Generating XML reports...
2022-12-01T11:12:12.6987179Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105241.xml
2022-12-01T11:12:12.6987565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6987724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6988108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6988302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6988322Z 
2022-12-01T11:12:12.6988433Z Running tests...
2022-12-01T11:12:12.6988698Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6989017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6989694Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6989728Z 
2022-12-01T11:12:12.6990004Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6990121Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6990140Z 
2022-12-01T11:12:12.6990231Z OK (skipped=1)
2022-12-01T11:12:12.6990251Z 
2022-12-01T11:12:12.6990377Z Generating XML reports...
2022-12-01T11:12:12.6990832Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105243.xml
2022-12-01T11:12:12.6991203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6991379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6991760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6992039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6992059Z 
2022-12-01T11:12:12.6992173Z Running tests...
2022-12-01T11:12:12.6992420Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6992735Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6993146Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6993166Z 
2022-12-01T11:12:12.6993432Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6993545Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6993565Z 
2022-12-01T11:12:12.6993673Z OK (skipped=1)
2022-12-01T11:12:12.6993693Z 
2022-12-01T11:12:12.6993818Z Generating XML reports...
2022-12-01T11:12:12.6994271Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105246.xml
2022-12-01T11:12:12.6994653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6994812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6995194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6995390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6995411Z 
2022-12-01T11:12:12.6995522Z Running tests...
2022-12-01T11:12:12.6995787Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6996105Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6996526Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6996546Z 
2022-12-01T11:12:12.6996812Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6996931Z Ran 1 test in 0.002s
2022-12-01T11:12:12.6997010Z 
2022-12-01T11:12:12.6997107Z OK (skipped=1)
2022-12-01T11:12:12.6997144Z 
2022-12-01T11:12:12.6997249Z Generating XML reports...
2022-12-01T11:12:12.6997705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105248.xml
2022-12-01T11:12:12.6998083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.6998261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.6998646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.6998838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.6998858Z 
2022-12-01T11:12:12.6998968Z Running tests...
2022-12-01T11:12:12.6999238Z ----------------------------------------------------------------------
2022-12-01T11:12:12.6999539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.6999952Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.6999971Z 
2022-12-01T11:12:12.7000231Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7000347Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7000366Z 
2022-12-01T11:12:12.7000474Z OK (skipped=1)
2022-12-01T11:12:12.7000493Z 
2022-12-01T11:12:12.7000619Z Generating XML reports...
2022-12-01T11:12:12.7001069Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105250.xml
2022-12-01T11:12:12.7001442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7001676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7002052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7002246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7002266Z 
2022-12-01T11:12:12.7002377Z Running tests...
2022-12-01T11:12:12.7002641Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7002956Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7003359Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7003379Z 
2022-12-01T11:12:12.7003641Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7003753Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7003772Z 
2022-12-01T11:12:12.7003881Z OK (skipped=1)
2022-12-01T11:12:12.7003904Z 
2022-12-01T11:12:12.7004009Z Generating XML reports...
2022-12-01T11:12:12.7004462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105252.xml
2022-12-01T11:12:12.7004840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7005019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7005395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7005590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7005610Z 
2022-12-01T11:12:12.7005721Z Running tests...
2022-12-01T11:12:12.7005985Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7006282Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7006680Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7006701Z 
2022-12-01T11:12:12.7006977Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7007090Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7007108Z 
2022-12-01T11:12:12.7007219Z OK (skipped=1)
2022-12-01T11:12:12.7007238Z 
2022-12-01T11:12:12.7007363Z Generating XML reports...
2022-12-01T11:12:12.7007815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105254.xml
2022-12-01T11:12:12.7008188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7008366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7008731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7008934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7008954Z 
2022-12-01T11:12:12.7009065Z Running tests...
2022-12-01T11:12:12.7009332Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7009649Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7010049Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7010069Z 
2022-12-01T11:12:12.7010334Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7010449Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7010468Z 
2022-12-01T11:12:12.7010577Z OK (skipped=1)
2022-12-01T11:12:12.7010596Z 
2022-12-01T11:12:12.7010701Z Generating XML reports...
2022-12-01T11:12:12.7011151Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105256.xml
2022-12-01T11:12:12.7011584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7011762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7012142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7012333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7012353Z 
2022-12-01T11:12:12.7012463Z Running tests...
2022-12-01T11:12:12.7012728Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7013024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7013439Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7013463Z 
2022-12-01T11:12:12.7013724Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7013840Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7013860Z 
2022-12-01T11:12:12.7013971Z OK (skipped=1)
2022-12-01T11:12:12.7013989Z 
2022-12-01T11:12:12.7014117Z Generating XML reports...
2022-12-01T11:12:12.7014569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105258.xml
2022-12-01T11:12:12.7014943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7015120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7015482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7015677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7015696Z 
2022-12-01T11:12:12.7015808Z Running tests...
2022-12-01T11:12:12.7016131Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7016458Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7016853Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7016873Z 
2022-12-01T11:12:12.7017135Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7017249Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7017268Z 
2022-12-01T11:12:12.7017377Z OK (skipped=1)
2022-12-01T11:12:12.7017396Z 
2022-12-01T11:12:12.7017501Z Generating XML reports...
2022-12-01T11:12:12.7017946Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105300.xml
2022-12-01T11:12:12.7018320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7018502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7018887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7019081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7019100Z 
2022-12-01T11:12:12.7019209Z Running tests...
2022-12-01T11:12:12.7019476Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7019791Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7020065Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7020086Z 
2022-12-01T11:12:12.7020350Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7020463Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7020532Z 
2022-12-01T11:12:12.7020644Z OK (skipped=1)
2022-12-01T11:12:12.7020662Z 
2022-12-01T11:12:12.7020792Z Generating XML reports...
2022-12-01T11:12:12.7021240Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105302.xml
2022-12-01T11:12:12.7021611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7021788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7022167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7022340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7022359Z 
2022-12-01T11:12:12.7022467Z Running tests...
2022-12-01T11:12:12.7022729Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7023041Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7023325Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7023345Z 
2022-12-01T11:12:12.7023611Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7023727Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7023746Z 
2022-12-01T11:12:12.7023854Z OK (skipped=1)
2022-12-01T11:12:12.7023872Z 
2022-12-01T11:12:12.7023977Z Generating XML reports...
2022-12-01T11:12:12.7024427Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105304.xml
2022-12-01T11:12:12.7024799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7024974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7025357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7025613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7025634Z 
2022-12-01T11:12:12.7025746Z Running tests...
2022-12-01T11:12:12.7026008Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7026323Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7026579Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7026618Z 
2022-12-01T11:12:12.7026861Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7026974Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7026993Z 
2022-12-01T11:12:12.7027101Z OK (skipped=1)
2022-12-01T11:12:12.7027119Z 
2022-12-01T11:12:12.7027244Z Generating XML reports...
2022-12-01T11:12:12.7027701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105306.xml
2022-12-01T11:12:12.7028084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7028262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7028646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7028821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7028841Z 
2022-12-01T11:12:12.7028951Z Running tests...
2022-12-01T11:12:12.7029217Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7029762Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7030053Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7030149Z 
2022-12-01T11:12:12.7030435Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7030549Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7030568Z 
2022-12-01T11:12:12.7030679Z OK (skipped=1)
2022-12-01T11:12:12.7030698Z 
2022-12-01T11:12:12.7030822Z Generating XML reports...
2022-12-01T11:12:12.7031255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105309.xml
2022-12-01T11:12:12.7031627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7031806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7032188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7032377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7032396Z 
2022-12-01T11:12:12.7032508Z Running tests...
2022-12-01T11:12:12.7032772Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7033091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7033346Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7033386Z 
2022-12-01T11:12:12.7033630Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7033741Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7033761Z 
2022-12-01T11:12:12.7033869Z OK (skipped=1)
2022-12-01T11:12:12.7033888Z 
2022-12-01T11:12:12.7034013Z Generating XML reports...
2022-12-01T11:12:12.7034463Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105311.xml
2022-12-01T11:12:12.7034835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7035018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7035460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7035643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7035681Z 
2022-12-01T11:12:12.7035772Z Running tests...
2022-12-01T11:12:12.7036039Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7036352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7036622Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7036642Z 
2022-12-01T11:12:12.7036904Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7037017Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7037036Z 
2022-12-01T11:12:12.7037150Z OK (skipped=1)
2022-12-01T11:12:12.7037168Z 
2022-12-01T11:12:12.7037297Z Generating XML reports...
2022-12-01T11:12:12.7037729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105313.xml
2022-12-01T11:12:12.7038100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7038277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7038657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7038849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7038868Z 
2022-12-01T11:12:12.7038978Z Running tests...
2022-12-01T11:12:12.7039243Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7039559Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7039865Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7039904Z 
2022-12-01T11:12:12.7040149Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7040262Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7040281Z 
2022-12-01T11:12:12.7040388Z OK (skipped=1)
2022-12-01T11:12:12.7040407Z 
2022-12-01T11:12:12.7040531Z Generating XML reports...
2022-12-01T11:12:12.7040974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105315.xml
2022-12-01T11:12:12.7041344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7041520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7041901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7042086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7042126Z 
2022-12-01T11:12:12.7042216Z Running tests...
2022-12-01T11:12:12.7042480Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7042793Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7043067Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7043086Z 
2022-12-01T11:12:12.7043350Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7043462Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7043482Z 
2022-12-01T11:12:12.7043591Z OK (skipped=1)
2022-12-01T11:12:12.7043609Z 
2022-12-01T11:12:12.7043735Z Generating XML reports...
2022-12-01T11:12:12.7044168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105317.xml
2022-12-01T11:12:12.7044589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7044772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7045153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7045344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7045364Z 
2022-12-01T11:12:12.7045472Z Running tests...
2022-12-01T11:12:12.7045735Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7046050Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7046318Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7046337Z 
2022-12-01T11:12:12.7046588Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7046704Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7046723Z 
2022-12-01T11:12:12.7046831Z OK (skipped=1)
2022-12-01T11:12:12.7046850Z 
2022-12-01T11:12:12.7046974Z Generating XML reports...
2022-12-01T11:12:12.7047422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105319.xml
2022-12-01T11:12:12.7047793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7047973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7048353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7048524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7048562Z 
2022-12-01T11:12:12.7048653Z Running tests...
2022-12-01T11:12:12.7048972Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7049289Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7049549Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7049568Z 
2022-12-01T11:12:12.7049826Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7049938Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7049957Z 
2022-12-01T11:12:12.7050065Z OK (skipped=1)
2022-12-01T11:12:12.7050084Z 
2022-12-01T11:12:12.7050208Z Generating XML reports...
2022-12-01T11:12:12.7050635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105321.xml
2022-12-01T11:12:12.7051011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7051191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7051580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7051772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7051791Z 
2022-12-01T11:12:12.7051900Z Running tests...
2022-12-01T11:12:12.7052165Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7052476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7052736Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7052756Z 
2022-12-01T11:12:12.7053000Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7053111Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7053130Z 
2022-12-01T11:12:12.7053238Z OK (skipped=1)
2022-12-01T11:12:12.7053261Z 
2022-12-01T11:12:12.7053385Z Generating XML reports...
2022-12-01T11:12:12.7053880Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105323.xml
2022-12-01T11:12:12.7054263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7054441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7054824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7055016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7055036Z 
2022-12-01T11:12:12.7055127Z Running tests...
2022-12-01T11:12:12.7055395Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7055710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7055995Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-12-01T11:12:12.7056014Z 
2022-12-01T11:12:12.7056278Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7056390Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7056410Z 
2022-12-01T11:12:12.7056517Z OK (skipped=1)
2022-12-01T11:12:12.7056536Z 
2022-12-01T11:12:12.7056663Z Generating XML reports...
2022-12-01T11:12:12.7057093Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105325.xml
2022-12-01T11:12:12.7057467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7057641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7058021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7058287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7058309Z 
2022-12-01T11:12:12.7058419Z Running tests...
2022-12-01T11:12:12.7058683Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7058996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7059287Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2022-12-01T11:12:12.7059307Z 
2022-12-01T11:12:12.7059549Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7059664Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7059683Z 
2022-12-01T11:12:12.7059790Z OK (skipped=1)
2022-12-01T11:12:12.7059809Z 
2022-12-01T11:12:12.7059934Z Generating XML reports...
2022-12-01T11:12:12.7060385Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105327.xml
2022-12-01T11:12:12.7060763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7060942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7061320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7061511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7061530Z 
2022-12-01T11:12:12.7061620Z Running tests...
2022-12-01T11:12:12.7061884Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7062201Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7062473Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7062492Z 
2022-12-01T11:12:12.7062756Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7062873Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7062939Z 
2022-12-01T11:12:12.7063055Z OK (skipped=1)
2022-12-01T11:12:12.7063075Z 
2022-12-01T11:12:12.7063199Z Generating XML reports...
2022-12-01T11:12:12.7063648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105330.xml
2022-12-01T11:12:12.7064000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7064177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7064554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7064745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7064763Z 
2022-12-01T11:12:12.7064875Z Running tests...
2022-12-01T11:12:12.7065146Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7065463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7065733Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7065940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75618
2022-12-01T11:12:12.7066162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75619
2022-12-01T11:12:12.7066535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7066714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7067098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7067292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7067716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7067890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7068267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7068438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7068688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7068937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7069387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7070002Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7070249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7070480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7070583Z ok (6.964s)
2022-12-01T11:12:12.7070605Z 
2022-12-01T11:12:12.7070858Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7070971Z Ran 1 test in 6.965s
2022-12-01T11:12:12.7070990Z 
2022-12-01T11:12:12.7071084Z OK
2022-12-01T11:12:12.7071103Z 
2022-12-01T11:12:12.7071225Z Generating XML reports...
2022-12-01T11:12:12.7071678Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105332.xml
2022-12-01T11:12:12.7072053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7072234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7072702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7072906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7072926Z 
2022-12-01T11:12:12.7073017Z Running tests...
2022-12-01T11:12:12.7073286Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7073603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7073865Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7073884Z 
2022-12-01T11:12:12.7074149Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7074264Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7074283Z 
2022-12-01T11:12:12.7074394Z OK (skipped=1)
2022-12-01T11:12:12.7074413Z 
2022-12-01T11:12:12.7074539Z Generating XML reports...
2022-12-01T11:12:12.7074997Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105341.xml
2022-12-01T11:12:12.7075352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7075528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7075909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7076104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7076123Z 
2022-12-01T11:12:12.7076234Z Running tests...
2022-12-01T11:12:12.7076497Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7076813Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7077083Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7077165Z 
2022-12-01T11:12:12.7077442Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7077538Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7077557Z 
2022-12-01T11:12:12.7077667Z OK (skipped=1)
2022-12-01T11:12:12.7077686Z 
2022-12-01T11:12:12.7077811Z Generating XML reports...
2022-12-01T11:12:12.7078258Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105343.xml
2022-12-01T11:12:12.7078629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7078805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7079186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7079380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7079402Z 
2022-12-01T11:12:12.7079495Z Running tests...
2022-12-01T11:12:12.7079764Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7080078Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7080349Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-12-01T11:12:12.7080369Z 
2022-12-01T11:12:12.7080631Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7080738Z Ran 1 test in 0.004s
2022-12-01T11:12:12.7080757Z 
2022-12-01T11:12:12.7080862Z OK (skipped=1)
2022-12-01T11:12:12.7080880Z 
2022-12-01T11:12:12.7081004Z Generating XML reports...
2022-12-01T11:12:12.7081447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105345.xml
2022-12-01T11:12:12.7081797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7082020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7082420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7082608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7082627Z 
2022-12-01T11:12:12.7082737Z Running tests...
2022-12-01T11:12:12.7083004Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7083317Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7083574Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7083779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76298
2022-12-01T11:12:12.7083989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76299
2022-12-01T11:12:12.7084363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7084539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7084920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7085106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7085459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7085624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7085988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7086158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7086455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7086690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7087082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7087468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7087688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7087910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7088244Z STAGE:2022-12-01 10:53:53 76298:76298 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7088564Z STAGE:2022-12-01 10:53:53 76299:76299 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7088890Z STAGE:2022-12-01 10:53:53 76299:76299 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7089209Z STAGE:2022-12-01 10:53:53 76298:76298 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7089525Z STAGE:2022-12-01 10:53:53 76299:76299 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7089850Z STAGE:2022-12-01 10:53:53 76298:76298 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7090171Z STAGE:2022-12-01 10:53:53 76299:76299 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7090491Z STAGE:2022-12-01 10:53:53 76298:76298 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7090809Z STAGE:2022-12-01 10:53:53 76299:76299 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7091120Z STAGE:2022-12-01 10:53:53 76298:76298 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7091510Z STAGE:2022-12-01 10:53:53 76299:76299 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7091834Z STAGE:2022-12-01 10:53:53 76298:76298 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7091926Z ok (6.395s)
2022-12-01T11:12:12.7091946Z 
2022-12-01T11:12:12.7092200Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7092304Z Ran 1 test in 6.395s
2022-12-01T11:12:12.7092323Z 
2022-12-01T11:12:12.7092405Z OK
2022-12-01T11:12:12.7092424Z 
2022-12-01T11:12:12.7092537Z Generating XML reports...
2022-12-01T11:12:12.7092979Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105347.xml
2022-12-01T11:12:12.7093346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7093510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7093886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7094068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7094088Z 
2022-12-01T11:12:12.7094185Z Running tests...
2022-12-01T11:12:12.7094441Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7094745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7095004Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7095219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76673
2022-12-01T11:12:12.7095428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76674
2022-12-01T11:12:12.7095781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7096004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7096376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7096559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7096912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7097077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7097442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7097620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7097850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7098094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7098490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7098877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7099098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7099318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7099655Z STAGE:2022-12-01 10:54:01 76674:76674 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7099971Z STAGE:2022-12-01 10:54:01 76673:76673 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7100302Z STAGE:2022-12-01 10:54:01 76673:76673 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7100665Z STAGE:2022-12-01 10:54:01 76674:76674 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7100995Z STAGE:2022-12-01 10:54:02 76673:76673 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7101304Z STAGE:2022-12-01 10:54:02 76674:76674 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7101619Z STAGE:2022-12-01 10:54:02 76673:76673 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7101941Z STAGE:2022-12-01 10:54:02 76674:76674 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7102255Z STAGE:2022-12-01 10:54:02 76673:76673 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7102567Z STAGE:2022-12-01 10:54:02 76674:76674 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7102893Z STAGE:2022-12-01 10:54:02 76673:76673 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7103222Z STAGE:2022-12-01 10:54:02 76674:76674 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7103308Z ok (6.269s)
2022-12-01T11:12:12.7103328Z 
2022-12-01T11:12:12.7103582Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7103695Z Ran 1 test in 6.269s
2022-12-01T11:12:12.7103715Z 
2022-12-01T11:12:12.7103809Z OK
2022-12-01T11:12:12.7103827Z 
2022-12-01T11:12:12.7103952Z Generating XML reports...
2022-12-01T11:12:12.7104408Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105356.xml
2022-12-01T11:12:12.7104782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7104957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7105323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7105564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7105583Z 
2022-12-01T11:12:12.7105685Z Running tests...
2022-12-01T11:12:12.7105950Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7106262Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7106538Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7106761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77096
2022-12-01T11:12:12.7106981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77097
2022-12-01T11:12:12.7107355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7107520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7107888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7108059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7108434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7108619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7108995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7109183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7109651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7109888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7110370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7110786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7111015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7111243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7111581Z STAGE:2022-12-01 10:54:09 77096:77096 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7111912Z STAGE:2022-12-01 10:54:10 77097:77097 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7112248Z STAGE:2022-12-01 10:54:10 77097:77097 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7112575Z STAGE:2022-12-01 10:54:10 77096:77096 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7112893Z STAGE:2022-12-01 10:54:10 77096:77096 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7113209Z STAGE:2022-12-01 10:54:10 77097:77097 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7113535Z STAGE:2022-12-01 10:54:10 77096:77096 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7113863Z STAGE:2022-12-01 10:54:10 77097:77097 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7114182Z STAGE:2022-12-01 10:54:10 77096:77096 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7114490Z STAGE:2022-12-01 10:54:10 77097:77097 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7114811Z STAGE:2022-12-01 10:54:10 77096:77096 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7115127Z STAGE:2022-12-01 10:54:10 77097:77097 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7115290Z ok (6.023s)
2022-12-01T11:12:12.7115311Z 
2022-12-01T11:12:12.7115561Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7115666Z Ran 1 test in 6.023s
2022-12-01T11:12:12.7115685Z 
2022-12-01T11:12:12.7115769Z OK
2022-12-01T11:12:12.7115788Z 
2022-12-01T11:12:12.7115906Z Generating XML reports...
2022-12-01T11:12:12.7116353Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105404.xml
2022-12-01T11:12:12.7116716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7116885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7117261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7117452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7117472Z 
2022-12-01T11:12:12.7117566Z Running tests...
2022-12-01T11:12:12.7117821Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7118132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7118370Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:12:12.7118389Z 
2022-12-01T11:12:12.7118643Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7118745Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7118765Z 
2022-12-01T11:12:12.7118865Z OK (skipped=1)
2022-12-01T11:12:12.7118884Z 
2022-12-01T11:12:12.7118998Z Generating XML reports...
2022-12-01T11:12:12.7119431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105412.xml
2022-12-01T11:12:12.7119847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7120027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7120401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7120584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7120604Z 
2022-12-01T11:12:12.7120705Z Running tests...
2022-12-01T11:12:12.7120961Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7121268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7121518Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:12:12.7121538Z 
2022-12-01T11:12:12.7121784Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7121891Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7121911Z 
2022-12-01T11:12:12.7122014Z OK (skipped=1)
2022-12-01T11:12:12.7122033Z 
2022-12-01T11:12:12.7122151Z Generating XML reports...
2022-12-01T11:12:12.7122653Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105414.xml
2022-12-01T11:12:12.7123023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7123189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7123561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7123744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7123764Z 
2022-12-01T11:12:12.7123854Z Running tests...
2022-12-01T11:12:12.7124108Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7124481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7124727Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7124938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77630
2022-12-01T11:12:12.7125150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77631
2022-12-01T11:12:12.7125511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7125680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7126044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7126228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7126590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7126761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7127131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7127312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7127550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7127790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7128191Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7128576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7128805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7129071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7129171Z ok (6.858s)
2022-12-01T11:12:12.7129191Z 
2022-12-01T11:12:12.7129451Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7129555Z Ran 1 test in 6.858s
2022-12-01T11:12:12.7129574Z 
2022-12-01T11:12:12.7129654Z OK
2022-12-01T11:12:12.7129673Z 
2022-12-01T11:12:12.7129789Z Generating XML reports...
2022-12-01T11:12:12.7130222Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105417.xml
2022-12-01T11:12:12.7130589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7130760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7131133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7131323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7131343Z 
2022-12-01T11:12:12.7131441Z Running tests...
2022-12-01T11:12:12.7131696Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7132004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7132262Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7132466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77991
2022-12-01T11:12:12.7132674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77992
2022-12-01T11:12:12.7133038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7133256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7133631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7133812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7134169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7134336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7134693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7134878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7135116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7135350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7135750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7136139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7136358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7136577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7136667Z ok (6.689s)
2022-12-01T11:12:12.7136687Z 
2022-12-01T11:12:12.7136934Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7137035Z Ran 1 test in 6.689s
2022-12-01T11:12:12.7137054Z 
2022-12-01T11:12:12.7137137Z OK
2022-12-01T11:12:12.7137156Z 
2022-12-01T11:12:12.7137271Z Generating XML reports...
2022-12-01T11:12:12.7137714Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105426.xml
2022-12-01T11:12:12.7138125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7138307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7138685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7138866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7138885Z 
2022-12-01T11:12:12.7138975Z Running tests...
2022-12-01T11:12:12.7139234Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7139542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7139797Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:12:12.7139820Z 
2022-12-01T11:12:12.7140080Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7140184Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7140203Z 
2022-12-01T11:12:12.7140307Z OK (skipped=1)
2022-12-01T11:12:12.7140327Z 
2022-12-01T11:12:12.7140443Z Generating XML reports...
2022-12-01T11:12:12.7140873Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105434.xml
2022-12-01T11:12:12.7141238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7141404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7141776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7141960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7141979Z 
2022-12-01T11:12:12.7142134Z Running tests...
2022-12-01T11:12:12.7142393Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7142701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7142965Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7143167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78561
2022-12-01T11:12:12.7143378Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78562
2022-12-01T11:12:12.7143739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7143906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7144270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7144457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7144821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7144996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7145356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7145540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7145792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7146037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7146429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7146825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7147105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7147345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7147572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7147796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7148197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7148591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7148694Z ok (6.808s)
2022-12-01T11:12:12.7148715Z 
2022-12-01T11:12:12.7148977Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7149087Z Ran 1 test in 6.808s
2022-12-01T11:12:12.7149110Z 
2022-12-01T11:12:12.7149194Z OK
2022-12-01T11:12:12.7149213Z 
2022-12-01T11:12:12.7149326Z Generating XML reports...
2022-12-01T11:12:12.7150007Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105437.xml
2022-12-01T11:12:12.7150367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7150538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7150914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7151099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7151120Z 
2022-12-01T11:12:12.7151222Z Running tests...
2022-12-01T11:12:12.7151479Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7151916Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7152163Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:12:12.7152183Z 
2022-12-01T11:12:12.7152447Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7152542Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7152562Z 
2022-12-01T11:12:12.7152661Z OK (skipped=1)
2022-12-01T11:12:12.7152680Z 
2022-12-01T11:12:12.7152802Z Generating XML reports...
2022-12-01T11:12:12.7153242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105446.xml
2022-12-01T11:12:12.7153603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7153777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7154167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7154354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7154373Z 
2022-12-01T11:12:12.7154479Z Running tests...
2022-12-01T11:12:12.7154727Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7155035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7155297Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7155510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78999
2022-12-01T11:12:12.7155724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79000
2022-12-01T11:12:12.7156088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7156344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7156731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7156907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7157272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7157447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7157814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7158004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7158246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7158494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7158902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7159295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7159508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7159737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7159890Z skip: Skipped due to small world size. (3.910s)
2022-12-01T11:12:12.7159910Z 
2022-12-01T11:12:12.7160173Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7160277Z Ran 1 test in 3.911s
2022-12-01T11:12:12.7160297Z 
2022-12-01T11:12:12.7160402Z OK (skipped=1)
2022-12-01T11:12:12.7160422Z 
2022-12-01T11:12:12.7160592Z Generating XML reports...
2022-12-01T11:12:12.7161050Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105448.xml
2022-12-01T11:12:12.7161403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7161575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7161954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7162138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7162158Z 
2022-12-01T11:12:12.7162267Z Running tests...
2022-12-01T11:12:12.7162528Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7162843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7163122Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7163149Z 
2022-12-01T11:12:12.7163413Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7163507Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7163526Z 
2022-12-01T11:12:12.7163627Z OK (skipped=1)
2022-12-01T11:12:12.7163646Z 
2022-12-01T11:12:12.7163762Z Generating XML reports...
2022-12-01T11:12:12.7164210Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105454.xml
2022-12-01T11:12:12.7164574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7164751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7165121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7165315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7165335Z 
2022-12-01T11:12:12.7165487Z Running tests...
2022-12-01T11:12:12.7165745Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7166059Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7166358Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7166377Z 
2022-12-01T11:12:12.7166630Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7166740Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7166760Z 
2022-12-01T11:12:12.7166866Z OK (skipped=1)
2022-12-01T11:12:12.7166885Z 
2022-12-01T11:12:12.7167003Z Generating XML reports...
2022-12-01T11:12:12.7167452Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105456.xml
2022-12-01T11:12:12.7167828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7167987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7168366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7168558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7168577Z 
2022-12-01T11:12:12.7168680Z Running tests...
2022-12-01T11:12:12.7168942Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7169291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7169573Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7170335Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2022-12-01T11:12:12.7170404Z 
2022-12-01T11:12:12.7170680Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7170777Z Ran 1 test in 1.642s
2022-12-01T11:12:12.7170807Z 
2022-12-01T11:12:12.7170899Z OK (skipped=1)
2022-12-01T11:12:12.7170917Z 
2022-12-01T11:12:12.7171042Z Generating XML reports...
2022-12-01T11:12:12.7171486Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105458.xml
2022-12-01T11:12:12.7171858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7172032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7172414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7172606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7172625Z 
2022-12-01T11:12:12.7172735Z Running tests...
2022-12-01T11:12:12.7172982Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7173298Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7173588Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7173811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79663
2022-12-01T11:12:12.7174023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79664
2022-12-01T11:12:12.7174390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7174565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7174987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7175168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7175536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7175704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7176077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7176260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7176504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7176746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7177153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7177545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7177760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7177988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7178329Z STAGE:2022-12-01 10:55:07 79664:79664 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7178652Z STAGE:2022-12-01 10:55:07 79663:79663 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7178980Z STAGE:2022-12-01 10:55:09 79663:79663 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7179374Z STAGE:2022-12-01 10:55:09 79664:79664 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7179482Z ok (7.687s)
2022-12-01T11:12:12.7179502Z 
2022-12-01T11:12:12.7179766Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7179879Z Ran 1 test in 7.687s
2022-12-01T11:12:12.7179899Z 
2022-12-01T11:12:12.7179972Z OK
2022-12-01T11:12:12.7179991Z 
2022-12-01T11:12:12.7180115Z Generating XML reports...
2022-12-01T11:12:12.7180568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105502.xml
2022-12-01T11:12:12.7180945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7181123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7181505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7181703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7181725Z 
2022-12-01T11:12:12.7181835Z Running tests...
2022-12-01T11:12:12.7182081Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7182400Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7182699Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7182719Z 
2022-12-01T11:12:12.7182981Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7183097Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7183117Z 
2022-12-01T11:12:12.7183226Z OK (skipped=1)
2022-12-01T11:12:12.7183244Z 
2022-12-01T11:12:12.7183370Z Generating XML reports...
2022-12-01T11:12:12.7183822Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105512.xml
2022-12-01T11:12:12.7184243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7184409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7184794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7184985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7185004Z 
2022-12-01T11:12:12.7185114Z Running tests...
2022-12-01T11:12:12.7185381Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7185699Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7185994Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7186747Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.714s)
2022-12-01T11:12:12.7186773Z 
2022-12-01T11:12:12.7187036Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7187149Z Ran 1 test in 1.714s
2022-12-01T11:12:12.7187169Z 
2022-12-01T11:12:12.7187257Z OK (skipped=1)
2022-12-01T11:12:12.7187276Z 
2022-12-01T11:12:12.7187401Z Generating XML reports...
2022-12-01T11:12:12.7187854Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105514.xml
2022-12-01T11:12:12.7188227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7188405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7188851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7189042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7189061Z 
2022-12-01T11:12:12.7189171Z Running tests...
2022-12-01T11:12:12.7189654Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7189971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7190269Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7190291Z 
2022-12-01T11:12:12.7190554Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7190672Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7190692Z 
2022-12-01T11:12:12.7190800Z OK (skipped=1)
2022-12-01T11:12:12.7190819Z 
2022-12-01T11:12:12.7190947Z Generating XML reports...
2022-12-01T11:12:12.7191402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105518.xml
2022-12-01T11:12:12.7191781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7191939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7192319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7192512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7192531Z 
2022-12-01T11:12:12.7192642Z Running tests...
2022-12-01T11:12:12.7192907Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7193222Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7193513Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7193810Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80514
2022-12-01T11:12:12.7194038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80515
2022-12-01T11:12:12.7194400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7194574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7194956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7195153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7195517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7195691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7196070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7196259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7196488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7196734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7197144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7197547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7197781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7198010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7198245Z skip: Skipped due to small world size. (4.021s)
2022-12-01T11:12:12.7198266Z 
2022-12-01T11:12:12.7198534Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7198648Z Ran 1 test in 4.022s
2022-12-01T11:12:12.7198667Z 
2022-12-01T11:12:12.7198756Z OK (skipped=1)
2022-12-01T11:12:12.7198796Z 
2022-12-01T11:12:12.7198900Z Generating XML reports...
2022-12-01T11:12:12.7199355Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105520.xml
2022-12-01T11:12:12.7199730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7199907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7200289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7200486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7200509Z 
2022-12-01T11:12:12.7200620Z Running tests...
2022-12-01T11:12:12.7200887Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7201182Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7201470Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7201490Z 
2022-12-01T11:12:12.7201755Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7201871Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7201891Z 
2022-12-01T11:12:12.7202000Z OK (skipped=1)
2022-12-01T11:12:12.7202019Z 
2022-12-01T11:12:12.7202144Z Generating XML reports...
2022-12-01T11:12:12.7202590Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105526.xml
2022-12-01T11:12:12.7203013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7203198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7203566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7203757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7203777Z 
2022-12-01T11:12:12.7203885Z Running tests...
2022-12-01T11:12:12.7204150Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7204465Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7204766Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7204786Z 
2022-12-01T11:12:12.7205056Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7205174Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7205193Z 
2022-12-01T11:12:12.7205301Z OK (skipped=1)
2022-12-01T11:12:12.7205320Z 
2022-12-01T11:12:12.7205424Z Generating XML reports...
2022-12-01T11:12:12.7205878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105528.xml
2022-12-01T11:12:12.7206249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7206427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7206808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7207001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7207021Z 
2022-12-01T11:12:12.7207133Z Running tests...
2022-12-01T11:12:12.7207455Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7207756Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7208040Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7208261Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81001
2022-12-01T11:12:12.7208480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81002
2022-12-01T11:12:12.7208856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7209034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7209419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7209612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7209987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7210142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7210521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7210712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7210964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7211211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7211613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7212014Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7212301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7212539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7212624Z ok (7.033s)
2022-12-01T11:12:12.7212644Z 
2022-12-01T11:12:12.7212913Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7213026Z Ran 1 test in 7.033s
2022-12-01T11:12:12.7213046Z 
2022-12-01T11:12:12.7213138Z OK
2022-12-01T11:12:12.7213157Z 
2022-12-01T11:12:12.7213281Z Generating XML reports...
2022-12-01T11:12:12.7213733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105530.xml
2022-12-01T11:12:12.7214109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7214287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7214660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7214853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7214873Z 
2022-12-01T11:12:12.7214984Z Running tests...
2022-12-01T11:12:12.7215247Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7215563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7215857Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7216081Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81388
2022-12-01T11:12:12.7216302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81389
2022-12-01T11:12:12.7216680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7216896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7217271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7217448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7217827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7218020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7218400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7218591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7218837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7219075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7219483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7219878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7220108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7220339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7220441Z ok (6.797s)
2022-12-01T11:12:12.7220460Z 
2022-12-01T11:12:12.7220727Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7220842Z Ran 1 test in 6.797s
2022-12-01T11:12:12.7220861Z 
2022-12-01T11:12:12.7220954Z OK
2022-12-01T11:12:12.7220973Z 
2022-12-01T11:12:12.7221085Z Generating XML reports...
2022-12-01T11:12:12.7221598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105539.xml
2022-12-01T11:12:12.7221982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7222160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7222540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7222731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7222750Z 
2022-12-01T11:12:12.7222860Z Running tests...
2022-12-01T11:12:12.7223125Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7223421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7223730Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7223753Z 
2022-12-01T11:12:12.7224020Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7224139Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7224158Z 
2022-12-01T11:12:12.7224268Z OK (skipped=1)
2022-12-01T11:12:12.7224287Z 
2022-12-01T11:12:12.7224413Z Generating XML reports...
2022-12-01T11:12:12.7224864Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105548.xml
2022-12-01T11:12:12.7225239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7225419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7225784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7226032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7226054Z 
2022-12-01T11:12:12.7226162Z Running tests...
2022-12-01T11:12:12.7226432Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7226747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7227043Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7227264Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81839
2022-12-01T11:12:12.7227483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81840
2022-12-01T11:12:12.7227858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7228018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7228412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7228606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7228975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7229152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7229744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7229943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7230193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7230421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7230833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7231309Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7231549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7231790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7232019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7232262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7232663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7233058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7233147Z ok (6.977s)
2022-12-01T11:12:12.7233191Z 
2022-12-01T11:12:12.7233444Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7233559Z Ran 1 test in 6.977s
2022-12-01T11:12:12.7233579Z 
2022-12-01T11:12:12.7233671Z OK
2022-12-01T11:12:12.7233690Z 
2022-12-01T11:12:12.7233814Z Generating XML reports...
2022-12-01T11:12:12.7234267Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105550.xml
2022-12-01T11:12:12.7234641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7234820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7235205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7235379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7235481Z 
2022-12-01T11:12:12.7235579Z Running tests...
2022-12-01T11:12:12.7235850Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7236169Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7236468Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:12:12.7236489Z 
2022-12-01T11:12:12.7236754Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7236866Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7236885Z 
2022-12-01T11:12:12.7236994Z OK (skipped=1)
2022-12-01T11:12:12.7237013Z 
2022-12-01T11:12:12.7237137Z Generating XML reports...
2022-12-01T11:12:12.7237572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105600.xml
2022-12-01T11:12:12.7237953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7238131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7238507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7238699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7238718Z 
2022-12-01T11:12:12.7238829Z Running tests...
2022-12-01T11:12:12.7239097Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7239414Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7239687Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7239909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82263
2022-12-01T11:12:12.7240132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82264
2022-12-01T11:12:12.7240567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7240754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7241127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7241303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7241689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7241881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7242240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7242433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7242689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7242938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7243338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7243737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7243970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7244202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7244365Z skip: Skipped due to small world size. (4.059s)
2022-12-01T11:12:12.7244385Z 
2022-12-01T11:12:12.7244633Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7244799Z Ran 1 test in 4.059s
2022-12-01T11:12:12.7244822Z 
2022-12-01T11:12:12.7244930Z OK (skipped=1)
2022-12-01T11:12:12.7244951Z 
2022-12-01T11:12:12.7245077Z Generating XML reports...
2022-12-01T11:12:12.7245532Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105602.xml
2022-12-01T11:12:12.7245905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7246081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7246459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7246650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7246671Z 
2022-12-01T11:12:12.7246763Z Running tests...
2022-12-01T11:12:12.7247026Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7247348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7247617Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7247836Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82597
2022-12-01T11:12:12.7248054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82598
2022-12-01T11:12:12.7248429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7248605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7248971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7249165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7249583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7249764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7250143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7250334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7250581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7250826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7251229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7251609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7251847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7252077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7252320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7252564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7252963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7253357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7253460Z ok (6.632s)
2022-12-01T11:12:12.7253480Z 
2022-12-01T11:12:12.7253748Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7253896Z Ran 1 test in 6.633s
2022-12-01T11:12:12.7253915Z 
2022-12-01T11:12:12.7254007Z OK
2022-12-01T11:12:12.7254027Z 
2022-12-01T11:12:12.7254157Z Generating XML reports...
2022-12-01T11:12:12.7254615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105608.xml
2022-12-01T11:12:12.7254989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7255166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7255551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7255743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7255762Z 
2022-12-01T11:12:12.7255852Z Running tests...
2022-12-01T11:12:12.7256117Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7256432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7256700Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7256919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82986
2022-12-01T11:12:12.7257140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82987
2022-12-01T11:12:12.7257516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7257693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7258079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7258255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7258620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7258847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7259237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7259428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7259679Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7259927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7260329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7260706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7260941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7261180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7261335Z skip: Need at least 3 CUDA devices (3.955s)
2022-12-01T11:12:12.7261355Z 
2022-12-01T11:12:12.7261623Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7261740Z Ran 1 test in 3.955s
2022-12-01T11:12:12.7261760Z 
2022-12-01T11:12:12.7261872Z OK (skipped=1)
2022-12-01T11:12:12.7261892Z 
2022-12-01T11:12:12.7262020Z Generating XML reports...
2022-12-01T11:12:12.7262477Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105617.xml
2022-12-01T11:12:12.7262832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7263011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7263395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7263648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7263667Z 
2022-12-01T11:12:12.7263780Z Running tests...
2022-12-01T11:12:12.7264048Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7264368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7264623Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.003s)
2022-12-01T11:12:12.7264643Z 
2022-12-01T11:12:12.7264908Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7265003Z Ran 1 test in 0.003s
2022-12-01T11:12:12.7265022Z 
2022-12-01T11:12:12.7265130Z OK (skipped=1)
2022-12-01T11:12:12.7265148Z 
2022-12-01T11:12:12.7265272Z Generating XML reports...
2022-12-01T11:12:12.7265722Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105623.xml
2022-12-01T11:12:12.7266102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7266278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7266662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7266859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7266878Z 
2022-12-01T11:12:12.7266968Z Running tests...
2022-12-01T11:12:12.7267236Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7267548Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7267801Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-12-01T11:12:12.7267825Z 
2022-12-01T11:12:12.7268135Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7268255Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7268274Z 
2022-12-01T11:12:12.7268385Z OK (skipped=1)
2022-12-01T11:12:12.7268404Z 
2022-12-01T11:12:12.7268529Z Generating XML reports...
2022-12-01T11:12:12.7268984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105625.xml
2022-12-01T11:12:12.7269387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7269567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7270238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7270439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7270460Z 
2022-12-01T11:12:12.7270575Z Running tests...
2022-12-01T11:12:12.7270851Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7271168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7271422Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7271627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83470
2022-12-01T11:12:12.7271844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83471
2022-12-01T11:12:12.7272217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7272396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7272776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7272967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7273428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7273602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7273977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7274150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7274398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7274644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7275047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7275446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7275686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7275916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7276018Z ok (7.640s)
2022-12-01T11:12:12.7276038Z 
2022-12-01T11:12:12.7276305Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7276402Z Ran 1 test in 7.640s
2022-12-01T11:12:12.7276421Z 
2022-12-01T11:12:12.7276513Z OK
2022-12-01T11:12:12.7276532Z 
2022-12-01T11:12:12.7276656Z Generating XML reports...
2022-12-01T11:12:12.7277107Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105627.xml
2022-12-01T11:12:12.7277481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7277664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7278105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7278307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7278326Z 
2022-12-01T11:12:12.7278417Z Running tests...
2022-12-01T11:12:12.7278686Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7278999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7279269Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-12-01T11:12:12.7279288Z 
2022-12-01T11:12:12.7279551Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7279662Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7279682Z 
2022-12-01T11:12:12.7279790Z OK (skipped=1)
2022-12-01T11:12:12.7279813Z 
2022-12-01T11:12:12.7279939Z Generating XML reports...
2022-12-01T11:12:12.7280393Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105637.xml
2022-12-01T11:12:12.7280748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7280926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7281302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7281493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7281512Z 
2022-12-01T11:12:12.7281621Z Running tests...
2022-12-01T11:12:12.7281883Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7282198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7282527Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7282750Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83964
2022-12-01T11:12:12.7282950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83965
2022-12-01T11:12:12.7283323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7283500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7283885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7284076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7284440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7284614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7284996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7285169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7285415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7285662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7286063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7286459Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7286690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7286921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7287144Z skip: Skipped due to small world size. (4.119s)
2022-12-01T11:12:12.7287165Z 
2022-12-01T11:12:12.7287439Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7287535Z Ran 1 test in 4.119s
2022-12-01T11:12:12.7287554Z 
2022-12-01T11:12:12.7287662Z OK (skipped=1)
2022-12-01T11:12:12.7287681Z 
2022-12-01T11:12:12.7287809Z Generating XML reports...
2022-12-01T11:12:12.7288265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105639.xml
2022-12-01T11:12:12.7288639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7288817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7289197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7289394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7289416Z 
2022-12-01T11:12:12.7289525Z Running tests...
2022-12-01T11:12:12.7289775Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7290091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7290353Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2022-12-01T11:12:12.7290373Z 
2022-12-01T11:12:12.7290633Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7290748Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7290767Z 
2022-12-01T11:12:12.7290877Z OK (skipped=1)
2022-12-01T11:12:12.7290897Z 
2022-12-01T11:12:12.7291020Z Generating XML reports...
2022-12-01T11:12:12.7291469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105645.xml
2022-12-01T11:12:12.7291883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7292060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7292443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7292633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7292653Z 
2022-12-01T11:12:12.7292761Z Running tests...
2022-12-01T11:12:12.7293026Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7293340Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7293601Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7293822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84308
2022-12-01T11:12:12.7294032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84309
2022-12-01T11:12:12.7294407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7294586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7294967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7295158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7295526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7295704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7296083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7296256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7296588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7296843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7297250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7297652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7297884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7298114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7298275Z skip: Skipped due to small world size. (3.974s)
2022-12-01T11:12:12.7298295Z 
2022-12-01T11:12:12.7298566Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7298663Z Ran 1 test in 3.974s
2022-12-01T11:12:12.7298684Z 
2022-12-01T11:12:12.7298793Z OK (skipped=1)
2022-12-01T11:12:12.7298812Z 
2022-12-01T11:12:12.7298935Z Generating XML reports...
2022-12-01T11:12:12.7299391Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105647.xml
2022-12-01T11:12:12.7299766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7299943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7300324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7300517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7300536Z 
2022-12-01T11:12:12.7300645Z Running tests...
2022-12-01T11:12:12.7300948Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7301268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7301550Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T11:12:12.7301570Z 
2022-12-01T11:12:12.7301830Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7301945Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7301964Z 
2022-12-01T11:12:12.7302072Z OK (skipped=1)
2022-12-01T11:12:12.7302090Z 
2022-12-01T11:12:12.7302215Z Generating XML reports...
2022-12-01T11:12:12.7302668Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105654.xml
2022-12-01T11:12:12.7303044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7303206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7303591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7303785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7303804Z 
2022-12-01T11:12:12.7303913Z Running tests...
2022-12-01T11:12:12.7304178Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7304493Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7304774Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T11:12:12.7304794Z 
2022-12-01T11:12:12.7305064Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7305157Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7305196Z 
2022-12-01T11:12:12.7305286Z OK (skipped=1)
2022-12-01T11:12:12.7305308Z 
2022-12-01T11:12:12.7305432Z Generating XML reports...
2022-12-01T11:12:12.7305928Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105656.xml
2022-12-01T11:12:12.7306313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7306494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7306876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7307070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7307089Z 
2022-12-01T11:12:12.7307199Z Running tests...
2022-12-01T11:12:12.7307445Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7307761Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7308045Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2022-12-01T11:12:12.7308065Z 
2022-12-01T11:12:12.7308329Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7308442Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7308461Z 
2022-12-01T11:12:12.7308571Z OK (skipped=1)
2022-12-01T11:12:12.7308590Z 
2022-12-01T11:12:12.7308714Z Generating XML reports...
2022-12-01T11:12:12.7309164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105658.xml
2022-12-01T11:12:12.7309776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7309942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7310335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7310613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7310637Z 
2022-12-01T11:12:12.7310746Z Running tests...
2022-12-01T11:12:12.7311011Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7311327Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7311585Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T11:12:12.7311605Z 
2022-12-01T11:12:12.7311863Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7311958Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7311996Z 
2022-12-01T11:12:12.7312085Z OK (skipped=1)
2022-12-01T11:12:12.7312104Z 
2022-12-01T11:12:12.7312227Z Generating XML reports...
2022-12-01T11:12:12.7312677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105700.xml
2022-12-01T11:12:12.7313060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7313237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7313620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7313812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7313832Z 
2022-12-01T11:12:12.7313942Z Running tests...
2022-12-01T11:12:12.7314187Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7314502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7314766Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2022-12-01T11:12:12.7314785Z 
2022-12-01T11:12:12.7315046Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7315161Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7315242Z 
2022-12-01T11:12:12.7315358Z OK (skipped=1)
2022-12-01T11:12:12.7315378Z 
2022-12-01T11:12:12.7315501Z Generating XML reports...
2022-12-01T11:12:12.7315956Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105702.xml
2022-12-01T11:12:12.7316330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7316487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7316864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7317056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7317076Z 
2022-12-01T11:12:12.7317184Z Running tests...
2022-12-01T11:12:12.7317450Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7317773Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7318059Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7318284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85121
2022-12-01T11:12:12.7318484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85122
2022-12-01T11:12:12.7318856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7319031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7319412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7319605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7320032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7320206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7320583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7320775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7321005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7321251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7321657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7322059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7322299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7322528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7322768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7323011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7323416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7323797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7324043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.7324283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.7324727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7325134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7325236Z ok (4.041s)
2022-12-01T11:12:12.7325256Z 
2022-12-01T11:12:12.7325522Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7325637Z Ran 1 test in 4.042s
2022-12-01T11:12:12.7325656Z 
2022-12-01T11:12:12.7325753Z OK
2022-12-01T11:12:12.7325772Z 
2022-12-01T11:12:12.7325878Z Generating XML reports...
2022-12-01T11:12:12.7326332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105704.xml
2022-12-01T11:12:12.7326706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7326887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7327271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7327464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7327484Z 
2022-12-01T11:12:12.7327593Z Running tests...
2022-12-01T11:12:12.7327859Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7328158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7328430Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7328653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85393
2022-12-01T11:12:12.7328875Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85394
2022-12-01T11:12:12.7329312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7329486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7329864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7330057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7330423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7330581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7330956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7331146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7331393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7331646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7332049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7332450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7332683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7332917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7333006Z ok (5.843s)
2022-12-01T11:12:12.7333025Z 
2022-12-01T11:12:12.7333292Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7333411Z Ran 1 test in 5.843s
2022-12-01T11:12:12.7333430Z 
2022-12-01T11:12:12.7333523Z OK
2022-12-01T11:12:12.7333545Z 
2022-12-01T11:12:12.7333670Z Generating XML reports...
2022-12-01T11:12:12.7334170Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105710.xml
2022-12-01T11:12:12.7334552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7334730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7335099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7335296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7335315Z 
2022-12-01T11:12:12.7335425Z Running tests...
2022-12-01T11:12:12.7335688Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7336005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7336292Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7336515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85737
2022-12-01T11:12:12.7336735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85738
2022-12-01T11:12:12.7337108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7337267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7337651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7337842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7338208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7338453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7338836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7339026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7339275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7339501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7339905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7340303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7340535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7340769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7340933Z skip: Skipped due to small world size. (3.876s)
2022-12-01T11:12:12.7340955Z 
2022-12-01T11:12:12.7341223Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7341337Z Ran 1 test in 3.876s
2022-12-01T11:12:12.7341357Z 
2022-12-01T11:12:12.7341466Z OK (skipped=1)
2022-12-01T11:12:12.7341486Z 
2022-12-01T11:12:12.7341591Z Generating XML reports...
2022-12-01T11:12:12.7342046Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105718.xml
2022-12-01T11:12:12.7342421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7342597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7342980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7343221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7343241Z 
2022-12-01T11:12:12.7343355Z Running tests...
2022-12-01T11:12:12.7343620Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7343936Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7344186Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7344410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85960
2022-12-01T11:12:12.7344628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85961
2022-12-01T11:12:12.7345002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7345180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7345566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7345759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7346125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7346282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7346656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7346846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7347093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7347339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7347801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7348204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7348435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7348664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7348748Z ok (5.534s)
2022-12-01T11:12:12.7348768Z 
2022-12-01T11:12:12.7349035Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7349150Z Ran 1 test in 5.535s
2022-12-01T11:12:12.7349170Z 
2022-12-01T11:12:12.7349264Z OK
2022-12-01T11:12:12.7349283Z 
2022-12-01T11:12:12.7349633Z Generating XML reports...
2022-12-01T11:12:12.7350103Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105724.xml
2022-12-01T11:12:12.7350483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7350662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7351024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7351218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7351239Z 
2022-12-01T11:12:12.7351350Z Running tests...
2022-12-01T11:12:12.7351619Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7351940Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7352217Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7352440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86345
2022-12-01T11:12:12.7352748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86346
2022-12-01T11:12:12.7353143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7353304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7353691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7353886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7354253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7354431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7354809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7355008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7355264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7355495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7355900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7356301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7356534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7356765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7356869Z ok (3.996s)
2022-12-01T11:12:12.7356888Z 
2022-12-01T11:12:12.7357157Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7357344Z Ran 1 test in 3.996s
2022-12-01T11:12:12.7357367Z 
2022-12-01T11:12:12.7357461Z OK
2022-12-01T11:12:12.7357479Z 
2022-12-01T11:12:12.7357584Z Generating XML reports...
2022-12-01T11:12:12.7358041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105732.xml
2022-12-01T11:12:12.7358413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7358590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7358977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7359171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7359191Z 
2022-12-01T11:12:12.7359300Z Running tests...
2022-12-01T11:12:12.7359567Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7359893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7360163Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7360383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86605
2022-12-01T11:12:12.7360603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86606
2022-12-01T11:12:12.7360976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7361159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7361541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7361734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7362151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7362317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7362695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7362886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7363134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7363379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7363782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7364181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7364425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7364655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7364739Z ok (5.818s)
2022-12-01T11:12:12.7364759Z 
2022-12-01T11:12:12.7365024Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7365138Z Ran 1 test in 5.818s
2022-12-01T11:12:12.7365158Z 
2022-12-01T11:12:12.7365251Z OK
2022-12-01T11:12:12.7365269Z 
2022-12-01T11:12:12.7365394Z Generating XML reports...
2022-12-01T11:12:12.7365846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105738.xml
2022-12-01T11:12:12.7366220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7366399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7366823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7367015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7367035Z 
2022-12-01T11:12:12.7367143Z Running tests...
2022-12-01T11:12:12.7367411Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7367728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7368004Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7368225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86977
2022-12-01T11:12:12.7368444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86978
2022-12-01T11:12:12.7368819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7368987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7369417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7369608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7369975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7370149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7370523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7370715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7370961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7371187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7371644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7372055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7372284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7372512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7372619Z ok (5.706s)
2022-12-01T11:12:12.7372639Z 
2022-12-01T11:12:12.7372905Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7373019Z Ran 1 test in 5.706s
2022-12-01T11:12:12.7373038Z 
2022-12-01T11:12:12.7373194Z OK
2022-12-01T11:12:12.7373213Z 
2022-12-01T11:12:12.7373320Z Generating XML reports...
2022-12-01T11:12:12.7373782Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105746.xml
2022-12-01T11:12:12.7374157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7374334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7374713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7374907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7374926Z 
2022-12-01T11:12:12.7375036Z Running tests...
2022-12-01T11:12:12.7375299Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7375618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7375875Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7376158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87315
2022-12-01T11:12:12.7376377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87316
2022-12-01T11:12:12.7376754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7376930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7377309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7377503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7377868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7378027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7378412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7378610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7378857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7379103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7379504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7379907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7380140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7380371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7380459Z ok (4.111s)
2022-12-01T11:12:12.7380479Z 
2022-12-01T11:12:12.7380793Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7380913Z Ran 1 test in 4.111s
2022-12-01T11:12:12.7380933Z 
2022-12-01T11:12:12.7381026Z OK
2022-12-01T11:12:12.7381045Z 
2022-12-01T11:12:12.7381170Z Generating XML reports...
2022-12-01T11:12:12.7381629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105754.xml
2022-12-01T11:12:12.7382003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7382180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7382543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7382736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7382760Z 
2022-12-01T11:12:12.7382870Z Running tests...
2022-12-01T11:12:12.7383139Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7383461Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7383721Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7383741Z 
2022-12-01T11:12:12.7384003Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7384116Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7384135Z 
2022-12-01T11:12:12.7384245Z OK (skipped=1)
2022-12-01T11:12:12.7384264Z 
2022-12-01T11:12:12.7384369Z Generating XML reports...
2022-12-01T11:12:12.7384820Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105801.xml
2022-12-01T11:12:12.7385195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7385428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7385813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7386006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7386025Z 
2022-12-01T11:12:12.7386138Z Running tests...
2022-12-01T11:12:12.7386403Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7386720Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7386958Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7387712Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.678s)
2022-12-01T11:12:12.7387740Z 
2022-12-01T11:12:12.7388010Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7388123Z Ran 1 test in 1.678s
2022-12-01T11:12:12.7388143Z 
2022-12-01T11:12:12.7388253Z OK (skipped=1)
2022-12-01T11:12:12.7388272Z 
2022-12-01T11:12:12.7388397Z Generating XML reports...
2022-12-01T11:12:12.7388849Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105803.xml
2022-12-01T11:12:12.7389222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7389615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7389998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7390195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7390215Z 
2022-12-01T11:12:12.7390406Z Running tests...
2022-12-01T11:12:12.7390687Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7391000Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7391271Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7391291Z 
2022-12-01T11:12:12.7391552Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7391667Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7391686Z 
2022-12-01T11:12:12.7391797Z OK (skipped=1)
2022-12-01T11:12:12.7391816Z 
2022-12-01T11:12:12.7391921Z Generating XML reports...
2022-12-01T11:12:12.7392370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105807.xml
2022-12-01T11:12:12.7392751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7392927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7393310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7393502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7393521Z 
2022-12-01T11:12:12.7393632Z Running tests...
2022-12-01T11:12:12.7393897Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7394213Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7394465Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7394484Z 
2022-12-01T11:12:12.7394744Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7394929Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7394948Z 
2022-12-01T11:12:12.7395059Z OK (skipped=1)
2022-12-01T11:12:12.7395078Z 
2022-12-01T11:12:12.7395203Z Generating XML reports...
2022-12-01T11:12:12.7395658Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105809.xml
2022-12-01T11:12:12.7396032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7396209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7396590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7396767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7396786Z 
2022-12-01T11:12:12.7396895Z Running tests...
2022-12-01T11:12:12.7397162Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7397484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7397753Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2022-12-01T11:12:12.7397774Z 
2022-12-01T11:12:12.7398032Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7398145Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7398164Z 
2022-12-01T11:12:12.7398272Z OK (skipped=1)
2022-12-01T11:12:12.7398291Z 
2022-12-01T11:12:12.7398396Z Generating XML reports...
2022-12-01T11:12:12.7398845Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105811.xml
2022-12-01T11:12:12.7399221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7399399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7399830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7400030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7400049Z 
2022-12-01T11:12:12.7400162Z Running tests...
2022-12-01T11:12:12.7400431Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7400748Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7400995Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7401743Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.671s)
2022-12-01T11:12:12.7401768Z 
2022-12-01T11:12:12.7402036Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7402152Z Ran 1 test in 1.672s
2022-12-01T11:12:12.7402171Z 
2022-12-01T11:12:12.7402279Z OK (skipped=1)
2022-12-01T11:12:12.7402298Z 
2022-12-01T11:12:12.7402423Z Generating XML reports...
2022-12-01T11:12:12.7402871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105813.xml
2022-12-01T11:12:12.7403246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7403421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7403789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7403982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7404001Z 
2022-12-01T11:12:12.7404163Z Running tests...
2022-12-01T11:12:12.7404433Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7404744Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7405060Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7405806Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.662s)
2022-12-01T11:12:12.7405827Z 
2022-12-01T11:12:12.7406093Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7406207Z Ran 1 test in 1.662s
2022-12-01T11:12:12.7406226Z 
2022-12-01T11:12:12.7406336Z OK (skipped=1)
2022-12-01T11:12:12.7406358Z 
2022-12-01T11:12:12.7406465Z Generating XML reports...
2022-12-01T11:12:12.7406920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105817.xml
2022-12-01T11:12:12.7407297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7407476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7407859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7408048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7408068Z 
2022-12-01T11:12:12.7408180Z Running tests...
2022-12-01T11:12:12.7408444Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7408763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7409115Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7409878Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2022-12-01T11:12:12.7409899Z 
2022-12-01T11:12:12.7410163Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7410277Z Ran 1 test in 1.667s
2022-12-01T11:12:12.7410296Z 
2022-12-01T11:12:12.7410406Z OK (skipped=1)
2022-12-01T11:12:12.7410425Z 
2022-12-01T11:12:12.7410551Z Generating XML reports...
2022-12-01T11:12:12.7411005Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105821.xml
2022-12-01T11:12:12.7411379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7411566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7411951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7412125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7412145Z 
2022-12-01T11:12:12.7412257Z Running tests...
2022-12-01T11:12:12.7412525Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7412841Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7413106Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7413327Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88600
2022-12-01T11:12:12.7413601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88601
2022-12-01T11:12:12.7413980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7414139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7414525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7414721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7415090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7415266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7415642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7415834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7416092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7416339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7416722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7417118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7417353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7417581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7417840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0naxysh
2022-12-01T11:12:12.7418116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0naxysh/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7418434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmm0csn6
2022-12-01T11:12:12.7418714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmm0csn6/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7418954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7419175Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7419279Z ok (6.183s)
2022-12-01T11:12:12.7419299Z 
2022-12-01T11:12:12.7419570Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7419684Z Ran 1 test in 6.183s
2022-12-01T11:12:12.7419703Z 
2022-12-01T11:12:12.7419797Z OK
2022-12-01T11:12:12.7419817Z 
2022-12-01T11:12:12.7419941Z Generating XML reports...
2022-12-01T11:12:12.7420397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105825.xml
2022-12-01T11:12:12.7420779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7420938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7421323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7421515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7421535Z 
2022-12-01T11:12:12.7421644Z Running tests...
2022-12-01T11:12:12.7421908Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7422228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7422508Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7422787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88922
2022-12-01T11:12:12.7423068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88923
2022-12-01T11:12:12.7423450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7423627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7424010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7424205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7424570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7424745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7425123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7425323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7425551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7425797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7426201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7426602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7426835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7427065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7427324Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lz439jb
2022-12-01T11:12:12.7427646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lz439jb/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7427909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpidpgkj
2022-12-01T11:12:12.7428164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpidpgkj/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7428402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7428638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7428872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7429107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7429210Z ok (6.137s)
2022-12-01T11:12:12.7429230Z 
2022-12-01T11:12:12.7429823Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7429949Z Ran 1 test in 6.137s
2022-12-01T11:12:12.7429970Z 
2022-12-01T11:12:12.7430047Z OK
2022-12-01T11:12:12.7430068Z 
2022-12-01T11:12:12.7430196Z Generating XML reports...
2022-12-01T11:12:12.7430663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105833.xml
2022-12-01T11:12:12.7431037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7431216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7431604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7431799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7431819Z 
2022-12-01T11:12:12.7431929Z Running tests...
2022-12-01T11:12:12.7432196Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7432592Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7432867Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7433622Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-12-01T11:12:12.7433644Z 
2022-12-01T11:12:12.7433912Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7434027Z Ran 1 test in 1.612s
2022-12-01T11:12:12.7434047Z 
2022-12-01T11:12:12.7434157Z OK (skipped=1)
2022-12-01T11:12:12.7434175Z 
2022-12-01T11:12:12.7434300Z Generating XML reports...
2022-12-01T11:12:12.7434753Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105841.xml
2022-12-01T11:12:12.7435139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7435297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7435683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7435877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7435897Z 
2022-12-01T11:12:12.7436007Z Running tests...
2022-12-01T11:12:12.7436272Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7436588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7436880Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7437688Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.717s)
2022-12-01T11:12:12.7437710Z 
2022-12-01T11:12:12.7437984Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7438098Z Ran 1 test in 1.717s
2022-12-01T11:12:12.7438117Z 
2022-12-01T11:12:12.7438207Z OK (skipped=1)
2022-12-01T11:12:12.7438225Z 
2022-12-01T11:12:12.7438352Z Generating XML reports...
2022-12-01T11:12:12.7438805Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105845.xml
2022-12-01T11:12:12.7439182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7439360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7439751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7439951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7439971Z 
2022-12-01T11:12:12.7440080Z Running tests...
2022-12-01T11:12:12.7440349Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7440648Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7440938Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7441158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89637
2022-12-01T11:12:12.7441377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89638
2022-12-01T11:12:12.7441751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7442022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7442408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7442601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7442949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7443125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7443502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7443697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7443948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7444205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7444612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7445011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7445245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7445457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7445720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptujm2kij
2022-12-01T11:12:12.7445995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptujm2kij/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7446252Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpak6gqzmw
2022-12-01T11:12:12.7446573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpak6gqzmw/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7446802Z <class 'torch.nn.parameter.Parameter'>
2022-12-01T11:12:12.7446907Z ok (6.745s)
2022-12-01T11:12:12.7446927Z 
2022-12-01T11:12:12.7447198Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7447296Z Ran 1 test in 6.745s
2022-12-01T11:12:12.7447335Z 
2022-12-01T11:12:12.7447412Z OK
2022-12-01T11:12:12.7447432Z 
2022-12-01T11:12:12.7447560Z Generating XML reports...
2022-12-01T11:12:12.7448017Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105849.xml
2022-12-01T11:12:12.7448391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7448572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7448964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7449159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7449179Z 
2022-12-01T11:12:12.7449290Z Running tests...
2022-12-01T11:12:12.7449538Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7449853Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7450160Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7450384Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90028
2022-12-01T11:12:12.7450602Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90029
2022-12-01T11:12:12.7450976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7451212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7451604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7451780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7452146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7452322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7452697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7452886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7453134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7453378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7453792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7454193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7454406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7454636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7454895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52ktan_h
2022-12-01T11:12:12.7455171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52ktan_h/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7455431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm9od95r
2022-12-01T11:12:12.7455761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm9od95r/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7455872Z ok (6.775s)
2022-12-01T11:12:12.7455891Z 
2022-12-01T11:12:12.7456162Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7456280Z Ran 1 test in 6.775s
2022-12-01T11:12:12.7456300Z 
2022-12-01T11:12:12.7456374Z OK
2022-12-01T11:12:12.7456392Z 
2022-12-01T11:12:12.7456520Z Generating XML reports...
2022-12-01T11:12:12.7456975Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105858.xml
2022-12-01T11:12:12.7457347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7457524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7457905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7458101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7458123Z 
2022-12-01T11:12:12.7458233Z Running tests...
2022-12-01T11:12:12.7458482Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7458799Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7459063Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7459283Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90400
2022-12-01T11:12:12.7459501Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90401
2022-12-01T11:12:12.7459875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7460054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7460498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7460692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7461039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7461217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7461592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7461780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7462029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7462276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7462680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7463086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7463297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7463528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7463787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93ih2eft
2022-12-01T11:12:12.7464059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93ih2eft/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7464317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_q9qpor
2022-12-01T11:12:12.7464586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_q9qpor/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7464827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7465114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7465226Z ok (7.376s)
2022-12-01T11:12:12.7465245Z 
2022-12-01T11:12:12.7465501Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7465617Z Ran 1 test in 7.377s
2022-12-01T11:12:12.7465636Z 
2022-12-01T11:12:12.7465731Z OK
2022-12-01T11:12:12.7465750Z 
2022-12-01T11:12:12.7465876Z Generating XML reports...
2022-12-01T11:12:12.7466338Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105907.xml
2022-12-01T11:12:12.7466710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7466889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7467271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7467454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7467493Z 
2022-12-01T11:12:12.7467583Z Running tests...
2022-12-01T11:12:12.7467852Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7468168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7468460Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7468682Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90706
2022-12-01T11:12:12.7468902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90707
2022-12-01T11:12:12.7469321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7469559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7470167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7470369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7470747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7470921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7471296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7471485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7471732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7471978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7472391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7472774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7473004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7473233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7473493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpniu5uhef
2022-12-01T11:12:12.7473763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpniu5uhef/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7474016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzhbskowg
2022-12-01T11:12:12.7474287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzhbskowg/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7475143Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.7475258Z ok (6.285s)
2022-12-01T11:12:12.7475279Z 
2022-12-01T11:12:12.7475551Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7475647Z Ran 1 test in 6.285s
2022-12-01T11:12:12.7475666Z 
2022-12-01T11:12:12.7475763Z OK
2022-12-01T11:12:12.7475782Z 
2022-12-01T11:12:12.7475910Z Generating XML reports...
2022-12-01T11:12:12.7476373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105916.xml
2022-12-01T11:12:12.7476746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7476923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7477307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7477500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7477520Z 
2022-12-01T11:12:12.7477629Z Running tests...
2022-12-01T11:12:12.7477874Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7478193Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7478477Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7479300Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.724s)
2022-12-01T11:12:12.7479321Z 
2022-12-01T11:12:12.7479587Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7479701Z Ran 1 test in 1.724s
2022-12-01T11:12:12.7479720Z 
2022-12-01T11:12:12.7479828Z OK (skipped=1)
2022-12-01T11:12:12.7479848Z 
2022-12-01T11:12:12.7479973Z Generating XML reports...
2022-12-01T11:12:12.7480424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105925.xml
2022-12-01T11:12:12.7480779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7480968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7481355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7481550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7481570Z 
2022-12-01T11:12:12.7481680Z Running tests...
2022-12-01T11:12:12.7481945Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7482261Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7482592Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2022-12-01T11:12:12.7482611Z 
2022-12-01T11:12:12.7482871Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7482963Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7482983Z 
2022-12-01T11:12:12.7483094Z OK (skipped=1)
2022-12-01T11:12:12.7483117Z 
2022-12-01T11:12:12.7483242Z Generating XML reports...
2022-12-01T11:12:12.7483756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105929.xml
2022-12-01T11:12:12.7484141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7484317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7484701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7484898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7484918Z 
2022-12-01T11:12:12.7485028Z Running tests...
2022-12-01T11:12:12.7485276Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7485593Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7485851Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7486605Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.738s)
2022-12-01T11:12:12.7486625Z 
2022-12-01T11:12:12.7486887Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7486999Z Ran 1 test in 1.738s
2022-12-01T11:12:12.7487019Z 
2022-12-01T11:12:12.7487127Z OK (skipped=1)
2022-12-01T11:12:12.7487146Z 
2022-12-01T11:12:12.7487272Z Generating XML reports...
2022-12-01T11:12:12.7487721Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105931.xml
2022-12-01T11:12:12.7488077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7488315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7488702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7488896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7488915Z 
2022-12-01T11:12:12.7489024Z Running tests...
2022-12-01T11:12:12.7489290Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7489603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7489876Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7490097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91451
2022-12-01T11:12:12.7490300Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91452
2022-12-01T11:12:12.7490685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7490863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7491247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7491440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7491806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7491983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7492360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7492534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7492833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7493090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7493494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7493894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7494126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7494357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7494615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bd6bkih
2022-12-01T11:12:12.7494889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bd6bkih/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7495134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa7by1jas
2022-12-01T11:12:12.7495406Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa7by1jas/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7496201Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T11:12:12.7496536Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T11:12:12.7497326Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T11:12:12.7497716Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T11:12:12.7497820Z ok (7.233s)
2022-12-01T11:12:12.7497840Z 
2022-12-01T11:12:12.7498108Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7498232Z Ran 1 test in 7.233s
2022-12-01T11:12:12.7498252Z 
2022-12-01T11:12:12.7498345Z OK
2022-12-01T11:12:12.7498364Z 
2022-12-01T11:12:12.7498488Z Generating XML reports...
2022-12-01T11:12:12.7498920Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105935.xml
2022-12-01T11:12:12.7499293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7499472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7499862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7500059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7500078Z 
2022-12-01T11:12:12.7500188Z Running tests...
2022-12-01T11:12:12.7500456Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7500775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7501031Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7501784Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2022-12-01T11:12:12.7501825Z 
2022-12-01T11:12:12.7502075Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7502237Z Ran 1 test in 1.631s
2022-12-01T11:12:12.7502258Z 
2022-12-01T11:12:12.7502370Z OK (skipped=1)
2022-12-01T11:12:12.7502389Z 
2022-12-01T11:12:12.7502516Z Generating XML reports...
2022-12-01T11:12:12.7502971Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105944.xml
2022-12-01T11:12:12.7503342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7503522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7503906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7504082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7504101Z 
2022-12-01T11:12:12.7504212Z Running tests...
2022-12-01T11:12:12.7504484Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7504802Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7505076Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7505827Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.772s)
2022-12-01T11:12:12.7505848Z 
2022-12-01T11:12:12.7506113Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7506228Z Ran 1 test in 1.772s
2022-12-01T11:12:12.7506247Z 
2022-12-01T11:12:12.7506357Z OK (skipped=1)
2022-12-01T11:12:12.7506377Z 
2022-12-01T11:12:12.7506506Z Generating XML reports...
2022-12-01T11:12:12.7507000Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105948.xml
2022-12-01T11:12:12.7507378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7507554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7507935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7508128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7508148Z 
2022-12-01T11:12:12.7508259Z Running tests...
2022-12-01T11:12:12.7508522Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7508837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7509114Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7509344Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92128
2022-12-01T11:12:12.7509810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92129
2022-12-01T11:12:12.7510187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7510364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7510744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7510938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7511311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7511488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7511924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7512126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7512376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7512621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7513023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7513421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7513654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7513901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7514117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7514365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7514771Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7515170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7515432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzodbdlag
2022-12-01T11:12:12.7515708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzodbdlag/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7515963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__wd_ztm
2022-12-01T11:12:12.7516230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__wd_ztm/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7516539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7516759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7516995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7517227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7517337Z ok (7.658s)
2022-12-01T11:12:12.7517357Z 
2022-12-01T11:12:12.7517629Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7517742Z Ran 1 test in 7.659s
2022-12-01T11:12:12.7517762Z 
2022-12-01T11:12:12.7517857Z OK
2022-12-01T11:12:12.7517876Z 
2022-12-01T11:12:12.7518002Z Generating XML reports...
2022-12-01T11:12:12.7518458Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105952.xml
2022-12-01T11:12:12.7518823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7519003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7519387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7519582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7519601Z 
2022-12-01T11:12:12.7519710Z Running tests...
2022-12-01T11:12:12.7519977Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7520294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7520573Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7520776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92539
2022-12-01T11:12:12.7521045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92540
2022-12-01T11:12:12.7521434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7521611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7521999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7522190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7522559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7522735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7523116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7523298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7523547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7523794Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7524199Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7524600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7524832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7525112Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:12:12.7525337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7525664Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:12:12.7525905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmparc5qz7a
2022-12-01T11:12:12.7526179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmparc5qz7a/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7526437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy0g7wk9r
2022-12-01T11:12:12.7526707Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy0g7wk9r/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7526949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7527185Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7527413Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7527657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7527921Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:12:12.7528197Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:12:12.7528475Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:12:12.7528747Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:12:12.7528980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7529216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7529448Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7529724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7530009Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:12:12.7530268Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:12:12.7530547Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T11:12:12.7530819Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T11:12:12.7531055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7531292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7531530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7531768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7531874Z ok (8.087s)
2022-12-01T11:12:12.7531894Z 
2022-12-01T11:12:12.7532174Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7532271Z Ran 1 test in 8.087s
2022-12-01T11:12:12.7532292Z 
2022-12-01T11:12:12.7532391Z OK
2022-12-01T11:12:12.7532410Z 
2022-12-01T11:12:12.7532537Z Generating XML reports...
2022-12-01T11:12:12.7532997Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110002.xml
2022-12-01T11:12:12.7533373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7533556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7533941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7534191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7534211Z 
2022-12-01T11:12:12.7534324Z Running tests...
2022-12-01T11:12:12.7534575Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7534892Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7535166Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7535918Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.704s)
2022-12-01T11:12:12.7535938Z 
2022-12-01T11:12:12.7536206Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7536326Z Ran 1 test in 1.704s
2022-12-01T11:12:12.7536349Z 
2022-12-01T11:12:12.7536461Z OK (skipped=1)
2022-12-01T11:12:12.7536479Z 
2022-12-01T11:12:12.7536608Z Generating XML reports...
2022-12-01T11:12:12.7537062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110013.xml
2022-12-01T11:12:12.7537417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7537598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7537983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7538177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7538196Z 
2022-12-01T11:12:12.7538306Z Running tests...
2022-12-01T11:12:12.7538572Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7538938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7539217Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7539438Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93025
2022-12-01T11:12:12.7539637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93026
2022-12-01T11:12:12.7540013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7540191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7540572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7540772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7541147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7541323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7541697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7541868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7542117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7542362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7542766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7543168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7543456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7544003Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T11:12:12.7544235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7544768Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T11:12:12.7545031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0lzfton
2022-12-01T11:12:12.7545311Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0lzfton/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7545550Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu07c1u8c
2022-12-01T11:12:12.7545820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu07c1u8c/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7546061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7546301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7546579Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T11:12:12.7546858Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T11:12:12.7547210Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T11:12:12.7547520Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T11:12:12.7547851Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T11:12:12.7548161Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T11:12:12.7548488Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T11:12:12.7548805Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T11:12:12.7549046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7549283Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7549602Z ok (7.591s)
2022-12-01T11:12:12.7549625Z 
2022-12-01T11:12:12.7549910Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7550026Z Ran 1 test in 7.592s
2022-12-01T11:12:12.7550045Z 
2022-12-01T11:12:12.7550143Z OK
2022-12-01T11:12:12.7550163Z 
2022-12-01T11:12:12.7550268Z Generating XML reports...
2022-12-01T11:12:12.7550726Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110016.xml
2022-12-01T11:12:12.7551098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7551281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7551664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7551946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7551966Z 
2022-12-01T11:12:12.7552073Z Running tests...
2022-12-01T11:12:12.7552341Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7552660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7553037Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7553078Z 
2022-12-01T11:12:12.7553323Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7553439Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7553458Z 
2022-12-01T11:12:12.7553566Z OK (skipped=1)
2022-12-01T11:12:12.7553585Z 
2022-12-01T11:12:12.7553713Z Generating XML reports...
2022-12-01T11:12:12.7554173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110026.xml
2022-12-01T11:12:12.7554549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7554725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7555107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7555280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7555319Z 
2022-12-01T11:12:12.7555410Z Running tests...
2022-12-01T11:12:12.7555678Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7555994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7556447Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7556472Z 
2022-12-01T11:12:12.7556743Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7556857Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7556877Z 
2022-12-01T11:12:12.7556986Z OK (skipped=1)
2022-12-01T11:12:12.7557005Z 
2022-12-01T11:12:12.7557131Z Generating XML reports...
2022-12-01T11:12:12.7557564Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110028.xml
2022-12-01T11:12:12.7557940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7558118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7558504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7558706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7558725Z 
2022-12-01T11:12:12.7558834Z Running tests...
2022-12-01T11:12:12.7559098Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7559414Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7559864Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7559885Z 
2022-12-01T11:12:12.7560151Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7560245Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7560264Z 
2022-12-01T11:12:12.7560377Z OK (skipped=1)
2022-12-01T11:12:12.7560396Z 
2022-12-01T11:12:12.7560573Z Generating XML reports...
2022-12-01T11:12:12.7561030Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110030.xml
2022-12-01T11:12:12.7561404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7561579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7561962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7562159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7562178Z 
2022-12-01T11:12:12.7562267Z Running tests...
2022-12-01T11:12:12.7562531Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7562844Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7563304Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7563324Z 
2022-12-01T11:12:12.7563590Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7563703Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7563723Z 
2022-12-01T11:12:12.7563830Z OK (skipped=1)
2022-12-01T11:12:12.7563849Z 
2022-12-01T11:12:12.7563986Z Generating XML reports...
2022-12-01T11:12:12.7564437Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110033.xml
2022-12-01T11:12:12.7564810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7564968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7565354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7565594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7565615Z 
2022-12-01T11:12:12.7565727Z Running tests...
2022-12-01T11:12:12.7565991Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7566304Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7566750Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7566770Z 
2022-12-01T11:12:12.7567039Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7567153Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7567172Z 
2022-12-01T11:12:12.7567264Z OK (skipped=1)
2022-12-01T11:12:12.7567283Z 
2022-12-01T11:12:12.7567408Z Generating XML reports...
2022-12-01T11:12:12.7567867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110035.xml
2022-12-01T11:12:12.7568240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7568417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7568802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7568995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7569015Z 
2022-12-01T11:12:12.7569125Z Running tests...
2022-12-01T11:12:12.7569412Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7569728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7570235Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7570255Z 
2022-12-01T11:12:12.7570519Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7570633Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7570652Z 
2022-12-01T11:12:12.7570762Z OK (skipped=1)
2022-12-01T11:12:12.7570781Z 
2022-12-01T11:12:12.7570905Z Generating XML reports...
2022-12-01T11:12:12.7571363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110037.xml
2022-12-01T11:12:12.7571738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7571914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7572283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7572475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7572495Z 
2022-12-01T11:12:12.7572604Z Running tests...
2022-12-01T11:12:12.7572868Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7573184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7573630Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7573650Z 
2022-12-01T11:12:12.7573912Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7574025Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7574048Z 
2022-12-01T11:12:12.7574157Z OK (skipped=1)
2022-12-01T11:12:12.7574176Z 
2022-12-01T11:12:12.7574329Z Generating XML reports...
2022-12-01T11:12:12.7574796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110039.xml
2022-12-01T11:12:12.7575170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7575346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7575728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7575921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7575940Z 
2022-12-01T11:12:12.7576049Z Running tests...
2022-12-01T11:12:12.7576314Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7576626Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7577061Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7577100Z 
2022-12-01T11:12:12.7577348Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7577459Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7577479Z 
2022-12-01T11:12:12.7577587Z OK (skipped=1)
2022-12-01T11:12:12.7577606Z 
2022-12-01T11:12:12.7577731Z Generating XML reports...
2022-12-01T11:12:12.7578182Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110041.xml
2022-12-01T11:12:12.7578555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7578734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7579182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7579357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7579394Z 
2022-12-01T11:12:12.7579486Z Running tests...
2022-12-01T11:12:12.7579752Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7580070Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7580517Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7580537Z 
2022-12-01T11:12:12.7580798Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7580914Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7580933Z 
2022-12-01T11:12:12.7581044Z OK (skipped=1)
2022-12-01T11:12:12.7581066Z 
2022-12-01T11:12:12.7581190Z Generating XML reports...
2022-12-01T11:12:12.7581623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110043.xml
2022-12-01T11:12:12.7581996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7582174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7582555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7582752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7582771Z 
2022-12-01T11:12:12.7582880Z Running tests...
2022-12-01T11:12:12.7583144Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7583463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7583990Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7584012Z 
2022-12-01T11:12:12.7584285Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7584380Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7584400Z 
2022-12-01T11:12:12.7584509Z OK (skipped=1)
2022-12-01T11:12:12.7584528Z 
2022-12-01T11:12:12.7584652Z Generating XML reports...
2022-12-01T11:12:12.7585104Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110045.xml
2022-12-01T11:12:12.7585477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7585657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7586046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7586240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7586260Z 
2022-12-01T11:12:12.7586370Z Running tests...
2022-12-01T11:12:12.7586620Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7586935Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7587327Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7587347Z 
2022-12-01T11:12:12.7587610Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7587777Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7587796Z 
2022-12-01T11:12:12.7587905Z OK (skipped=1)
2022-12-01T11:12:12.7587927Z 
2022-12-01T11:12:12.7588052Z Generating XML reports...
2022-12-01T11:12:12.7588504Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110047.xml
2022-12-01T11:12:12.7588876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7589035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7589636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7589838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7589859Z 
2022-12-01T11:12:12.7589968Z Running tests...
2022-12-01T11:12:12.7590238Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7590560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7590952Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2022-12-01T11:12:12.7590972Z 
2022-12-01T11:12:12.7591237Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7591349Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7591369Z 
2022-12-01T11:12:12.7591459Z OK (skipped=1)
2022-12-01T11:12:12.7591477Z 
2022-12-01T11:12:12.7591602Z Generating XML reports...
2022-12-01T11:12:12.7592056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110049.xml
2022-12-01T11:12:12.7592432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7592608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7593065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7593268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7593288Z 
2022-12-01T11:12:12.7593397Z Running tests...
2022-12-01T11:12:12.7593646Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7593963Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7594232Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7594983Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s)
2022-12-01T11:12:12.7595006Z 
2022-12-01T11:12:12.7595273Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7595386Z Ran 1 test in 1.667s
2022-12-01T11:12:12.7595405Z 
2022-12-01T11:12:12.7595515Z OK (skipped=1)
2022-12-01T11:12:12.7595534Z 
2022-12-01T11:12:12.7595659Z Generating XML reports...
2022-12-01T11:12:12.7596110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110051.xml
2022-12-01T11:12:12.7596481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7596639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7597023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7597215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7597295Z 
2022-12-01T11:12:12.7597412Z Running tests...
2022-12-01T11:12:12.7597682Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7597997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7598254Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7598476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94542
2022-12-01T11:12:12.7598678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94543
2022-12-01T11:12:12.7599056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7599231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7599614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7599812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7600187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7600363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7600741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7600930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7601159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7601407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7601814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7602276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7602518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7602748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7603008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4af2ify
2022-12-01T11:12:12.7603284Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4af2ify/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7603541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp15kjjit_
2022-12-01T11:12:12.7603791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp15kjjit_/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7603897Z ok (6.559s)
2022-12-01T11:12:12.7603916Z 
2022-12-01T11:12:12.7604196Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7604314Z Ran 1 test in 6.559s
2022-12-01T11:12:12.7604333Z 
2022-12-01T11:12:12.7604429Z OK
2022-12-01T11:12:12.7604448Z 
2022-12-01T11:12:12.7604573Z Generating XML reports...
2022-12-01T11:12:12.7605029Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110055.xml
2022-12-01T11:12:12.7605406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7605565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7605946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7606138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7606157Z 
2022-12-01T11:12:12.7606266Z Running tests...
2022-12-01T11:12:12.7606534Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7606913Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7607189Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7607412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94871
2022-12-01T11:12:12.7607631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94872
2022-12-01T11:12:12.7607984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7608159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7608540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7608732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7609112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7609286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7609662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7609853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7610079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7610325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7610732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7611135Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7611429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7611665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7611924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvaz5793m
2022-12-01T11:12:12.7612195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvaz5793m/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7612450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqxlf0n_r
2022-12-01T11:12:12.7612704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqxlf0n_r/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7612944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7613181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7613593Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpvaz5793m'>
2022-12-01T11:12:12.7613766Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:12:12.7614169Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqxlf0n_r'>
2022-12-01T11:12:12.7614335Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:12:12.7614439Z ok (7.502s)
2022-12-01T11:12:12.7614459Z 
2022-12-01T11:12:12.7614726Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7614823Z Ran 1 test in 7.503s
2022-12-01T11:12:12.7614842Z 
2022-12-01T11:12:12.7614936Z OK
2022-12-01T11:12:12.7614954Z 
2022-12-01T11:12:12.7615081Z Generating XML reports...
2022-12-01T11:12:12.7615536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110104.xml
2022-12-01T11:12:12.7615907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7616143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7616528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7616724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7616743Z 
2022-12-01T11:12:12.7616834Z Running tests...
2022-12-01T11:12:12.7617104Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7617423Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7617705Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.006s)
2022-12-01T11:12:12.7617724Z 
2022-12-01T11:12:12.7617986Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7618103Z Ran 1 test in 0.006s
2022-12-01T11:12:12.7618126Z 
2022-12-01T11:12:12.7618237Z OK (skipped=1)
2022-12-01T11:12:12.7618256Z 
2022-12-01T11:12:12.7618384Z Generating XML reports...
2022-12-01T11:12:12.7618835Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110114.xml
2022-12-01T11:12:12.7619192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7619368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7619749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7619941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7619962Z 
2022-12-01T11:12:12.7620072Z Running tests...
2022-12-01T11:12:12.7620338Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7620655Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7620968Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7621177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95378
2022-12-01T11:12:12.7621398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95379
2022-12-01T11:12:12.7621775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7621953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7622337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7622529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7622893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7623077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7623444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7623638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7623887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7624136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7624536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7624935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7625149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7625437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7625696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgxvd1dnt
2022-12-01T11:12:12.7625969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgxvd1dnt/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7626222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1gj_i1zp
2022-12-01T11:12:12.7626489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1gj_i1zp/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7626725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7626958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7627042Z ok (7.383s)
2022-12-01T11:12:12.7627083Z 
2022-12-01T11:12:12.7627336Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7627453Z Ran 1 test in 7.383s
2022-12-01T11:12:12.7627474Z 
2022-12-01T11:12:12.7627569Z OK
2022-12-01T11:12:12.7627588Z 
2022-12-01T11:12:12.7627711Z Generating XML reports...
2022-12-01T11:12:12.7628161Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110116.xml
2022-12-01T11:12:12.7628534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7628712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7629097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7629273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7629293Z 
2022-12-01T11:12:12.7629617Z Running tests...
2022-12-01T11:12:12.7629895Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7630286Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7630583Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7630806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95781
2022-12-01T11:12:12.7631023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95782
2022-12-01T11:12:12.7631404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7631565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7631949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7632142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7632517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7632692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7633070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7633259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7633508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7633752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7634134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7634532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7634835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7635064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7635304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7635546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7635949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7636341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7636583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.7636802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.7637200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7637598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7637860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3mc_apm
2022-12-01T11:12:12.7638134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3mc_apm/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7638392Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cddx48q
2022-12-01T11:12:12.7638661Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cddx48q/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7638764Z ok (5.825s)
2022-12-01T11:12:12.7638784Z 
2022-12-01T11:12:12.7639054Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7639152Z Ran 1 test in 5.825s
2022-12-01T11:12:12.7639171Z 
2022-12-01T11:12:12.7639310Z OK
2022-12-01T11:12:12.7639331Z 
2022-12-01T11:12:12.7639462Z Generating XML reports...
2022-12-01T11:12:12.7639913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110126.xml
2022-12-01T11:12:12.7640287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7640462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7640843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7641035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7641055Z 
2022-12-01T11:12:12.7641166Z Running tests...
2022-12-01T11:12:12.7641413Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7641737Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7642018Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7642241Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96177
2022-12-01T11:12:12.7642461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96178
2022-12-01T11:12:12.7642832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7643008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7643389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7643563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7643931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7644163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7644544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7644735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7644983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7645229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7645631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7646029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7646248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7646480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7646717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7646957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7647356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7647751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7647995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.7648231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.7648673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7649059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7649316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_z3umcap
2022-12-01T11:12:12.7649588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_z3umcap/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7649843Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqy2y0c2
2022-12-01T11:12:12.7650110Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqy2y0c2/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7650213Z ok (16.150s)
2022-12-01T11:12:12.7650233Z 
2022-12-01T11:12:12.7650503Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7650623Z Ran 1 test in 16.150s
2022-12-01T11:12:12.7650642Z 
2022-12-01T11:12:12.7650735Z OK
2022-12-01T11:12:12.7650757Z 
2022-12-01T11:12:12.7650862Z Generating XML reports...
2022-12-01T11:12:12.7651316Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110134.xml
2022-12-01T11:12:12.7651688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7651865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7652247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7652439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7652458Z 
2022-12-01T11:12:12.7652570Z Running tests...
2022-12-01T11:12:12.7652833Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7653188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7653502Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7653723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96647
2022-12-01T11:12:12.7653940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96648
2022-12-01T11:12:12.7654314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7654490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7654869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7655062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7655431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7655595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7655972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7656163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7656411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7656660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7657063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7657460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7657699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7657973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7658220Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpojhiwbr0
2022-12-01T11:12:12.7658491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpojhiwbr0/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7658745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp35xukxfg
2022-12-01T11:12:12.7659017Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp35xukxfg/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7659120Z ok (7.750s)
2022-12-01T11:12:12.7659140Z 
2022-12-01T11:12:12.7659408Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7659525Z Ran 1 test in 7.750s
2022-12-01T11:12:12.7659545Z 
2022-12-01T11:12:12.7659637Z OK
2022-12-01T11:12:12.7659660Z 
2022-12-01T11:12:12.7659766Z Generating XML reports...
2022-12-01T11:12:12.7660224Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110152.xml
2022-12-01T11:12:12.7660599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7660776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7661155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7661346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7661365Z 
2022-12-01T11:12:12.7661475Z Running tests...
2022-12-01T11:12:12.7661739Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7662035Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7662382Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7662602Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97082
2022-12-01T11:12:12.7662821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97083
2022-12-01T11:12:12.7663195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7663370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7663747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7663937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7664301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7664462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7664847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7665034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7665283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7665530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7665930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7666328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7666562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7666801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7667103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq49lkj_q
2022-12-01T11:12:12.7667383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq49lkj_q/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7667637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg00zzd6u
2022-12-01T11:12:12.7667905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg00zzd6u/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7668003Z ok (7.833s)
2022-12-01T11:12:12.7668022Z 
2022-12-01T11:12:12.7668293Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7668406Z Ran 1 test in 7.833s
2022-12-01T11:12:12.7668426Z 
2022-12-01T11:12:12.7668519Z OK
2022-12-01T11:12:12.7668539Z 
2022-12-01T11:12:12.7668643Z Generating XML reports...
2022-12-01T11:12:12.7669102Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110202.xml
2022-12-01T11:12:12.7669533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7669711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7670301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7670497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7670518Z 
2022-12-01T11:12:12.7670628Z Running tests...
2022-12-01T11:12:12.7670898Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7671216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7671456Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7671758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97581
2022-12-01T11:12:12.7671978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97582
2022-12-01T11:12:12.7672356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7672532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7672913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7673104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7673468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7673624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7674003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7674200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7674446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7674689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7675087Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7675485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7675715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7675943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7676184Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5tvul9yy
2022-12-01T11:12:12.7676531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5tvul9yy/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7676796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptv3ogffc
2022-12-01T11:12:12.7677064Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptv3ogffc/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7677168Z ok (7.464s)
2022-12-01T11:12:12.7677188Z 
2022-12-01T11:12:12.7677456Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7677573Z Ran 1 test in 7.465s
2022-12-01T11:12:12.7677592Z 
2022-12-01T11:12:12.7677686Z OK
2022-12-01T11:12:12.7677705Z 
2022-12-01T11:12:12.7677827Z Generating XML reports...
2022-12-01T11:12:12.7678263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110212.xml
2022-12-01T11:12:12.7678634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7678820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7679201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7679390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7679409Z 
2022-12-01T11:12:12.7679518Z Running tests...
2022-12-01T11:12:12.7679780Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7680090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7680333Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7680554Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97991
2022-12-01T11:12:12.7680770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97992
2022-12-01T11:12:12.7681204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7681380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7681763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7681953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7682318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7682491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7682854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7683040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7683291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7683537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7683938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7684335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7684567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7684796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7685035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2zbtwfbc
2022-12-01T11:12:12.7685308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2zbtwfbc/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7685613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk91zal15
2022-12-01T11:12:12.7685889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk91zal15/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7686671Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.7687451Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.7687695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7687934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7688036Z ok (7.349s)
2022-12-01T11:12:12.7688056Z 
2022-12-01T11:12:12.7688327Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7688443Z Ran 1 test in 7.349s
2022-12-01T11:12:12.7688462Z 
2022-12-01T11:12:12.7688556Z OK
2022-12-01T11:12:12.7688574Z 
2022-12-01T11:12:12.7688698Z Generating XML reports...
2022-12-01T11:12:12.7689141Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110222.xml
2022-12-01T11:12:12.7689571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7689750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7690135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7690329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7690349Z 
2022-12-01T11:12:12.7690461Z Running tests...
2022-12-01T11:12:12.7690725Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7691041Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7691305Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7692066Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.796s)
2022-12-01T11:12:12.7692104Z 
2022-12-01T11:12:12.7692347Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7692460Z Ran 1 test in 1.796s
2022-12-01T11:12:12.7692479Z 
2022-12-01T11:12:12.7692586Z OK (skipped=1)
2022-12-01T11:12:12.7692604Z 
2022-12-01T11:12:12.7692729Z Generating XML reports...
2022-12-01T11:12:12.7693180Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110232.xml
2022-12-01T11:12:12.7693551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7693734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7694161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7694341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7694361Z 
2022-12-01T11:12:12.7694470Z Running tests...
2022-12-01T11:12:12.7694734Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7695051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7695332Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7696077Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2022-12-01T11:12:12.7696101Z 
2022-12-01T11:12:12.7696372Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7696485Z Ran 1 test in 1.628s
2022-12-01T11:12:12.7696505Z 
2022-12-01T11:12:12.7696616Z OK (skipped=1)
2022-12-01T11:12:12.7696634Z 
2022-12-01T11:12:12.7696757Z Generating XML reports...
2022-12-01T11:12:12.7697187Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110236.xml
2022-12-01T11:12:12.7697563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7697739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7698121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7698309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7698373Z 
2022-12-01T11:12:12.7698487Z Running tests...
2022-12-01T11:12:12.7698758Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7699073Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7699334Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7699556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98694
2022-12-01T11:12:12.7699775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98695
2022-12-01T11:12:12.7700146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7700324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7700705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7700907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7701280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7701456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7701811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7702005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7702253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7702498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7702897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7703341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7703579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7703807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7704048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrwi4uvl
2022-12-01T11:12:12.7704325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrwi4uvl/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7704583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_o_7b8i
2022-12-01T11:12:12.7704844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_o_7b8i/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7705187Z STAGE:2022-12-01 11:02:45 98694:98694 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7705523Z STAGE:2022-12-01 11:02:45 98695:98695 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7705759Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7705996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7706334Z STAGE:2022-12-01 11:02:46 98694:98694 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7706647Z STAGE:2022-12-01 11:02:46 98695:98695 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7707430Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.7708281Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.7708620Z STAGE:2022-12-01 11:02:46 98694:98694 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7708927Z STAGE:2022-12-01 11:02:46 98695:98695 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7709263Z STAGE:2022-12-01 11:02:46 98694:98694 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7709823Z STAGE:2022-12-01 11:02:46 98695:98695 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7709931Z ok (6.762s)
2022-12-01T11:12:12.7709952Z 
2022-12-01T11:12:12.7710221Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7710333Z Ran 1 test in 6.762s
2022-12-01T11:12:12.7710352Z 
2022-12-01T11:12:12.7710444Z OK
2022-12-01T11:12:12.7710463Z 
2022-12-01T11:12:12.7710588Z Generating XML reports...
2022-12-01T11:12:12.7711025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110240.xml
2022-12-01T11:12:12.7711399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7711578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7712034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7712236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7712256Z 
2022-12-01T11:12:12.7712365Z Running tests...
2022-12-01T11:12:12.7712634Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7712950Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7713219Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7713423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99107
2022-12-01T11:12:12.7713642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99108
2022-12-01T11:12:12.7714014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7714199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7714583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7714777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7715143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7715315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7715668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7715856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7716104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7716349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7716826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7717226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7717459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7717691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7717950Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxkaudyj
2022-12-01T11:12:12.7718206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxkaudyj/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7718464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8h0w2vex
2022-12-01T11:12:12.7718733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8h0w2vex/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7718843Z ok (6.954s)
2022-12-01T11:12:12.7718863Z 
2022-12-01T11:12:12.7719131Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7719250Z Ran 1 test in 6.954s
2022-12-01T11:12:12.7719269Z 
2022-12-01T11:12:12.7719360Z OK
2022-12-01T11:12:12.7719379Z 
2022-12-01T11:12:12.7719502Z Generating XML reports...
2022-12-01T11:12:12.7719958Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110249.xml
2022-12-01T11:12:12.7720315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7720493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7720877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7721074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7721093Z 
2022-12-01T11:12:12.7721249Z Running tests...
2022-12-01T11:12:12.7721522Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7721840Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7722118Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7722924Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.681s)
2022-12-01T11:12:12.7722946Z 
2022-12-01T11:12:12.7723202Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7723316Z Ran 1 test in 1.681s
2022-12-01T11:12:12.7723340Z 
2022-12-01T11:12:12.7723449Z OK (skipped=1)
2022-12-01T11:12:12.7723471Z 
2022-12-01T11:12:12.7723597Z Generating XML reports...
2022-12-01T11:12:12.7724047Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110258.xml
2022-12-01T11:12:12.7724417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7724593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7724972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7725165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7725185Z 
2022-12-01T11:12:12.7725274Z Running tests...
2022-12-01T11:12:12.7725539Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7725956Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7726239Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7726460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99610
2022-12-01T11:12:12.7726678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99611
2022-12-01T11:12:12.7727052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7727229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7727596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7727789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7728156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7728335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7728713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7728905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7729156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7729403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7729806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7730190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7730421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7730696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7730959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz42amt40
2022-12-01T11:12:12.7731229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz42amt40/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7731483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwmx2s44
2022-12-01T11:12:12.7731752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwmx2s44/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7732673Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T11:12:12.7732794Z   warnings.warn(
2022-12-01T11:12:12.7733718Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T11:12:12.7733829Z   warnings.warn(
2022-12-01T11:12:12.7734052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7734282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7734383Z ok (6.194s)
2022-12-01T11:12:12.7734402Z 
2022-12-01T11:12:12.7734670Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7734836Z Ran 1 test in 6.194s
2022-12-01T11:12:12.7734855Z 
2022-12-01T11:12:12.7734948Z OK
2022-12-01T11:12:12.7734970Z 
2022-12-01T11:12:12.7735094Z Generating XML reports...
2022-12-01T11:12:12.7735550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110302.xml
2022-12-01T11:12:12.7735905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7736081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7736462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7736652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7736671Z 
2022-12-01T11:12:12.7736777Z Running tests...
2022-12-01T11:12:12.7737039Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7737359Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7737639Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7738388Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.614s)
2022-12-01T11:12:12.7738409Z 
2022-12-01T11:12:12.7738671Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7738764Z Ran 1 test in 1.614s
2022-12-01T11:12:12.7738782Z 
2022-12-01T11:12:12.7738889Z OK (skipped=1)
2022-12-01T11:12:12.7738908Z 
2022-12-01T11:12:12.7739033Z Generating XML reports...
2022-12-01T11:12:12.7739481Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110310.xml
2022-12-01T11:12:12.7739914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7740098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7740482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7740674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7740693Z 
2022-12-01T11:12:12.7740781Z Running tests...
2022-12-01T11:12:12.7741045Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7741356Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7741632Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7741854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100138
2022-12-01T11:12:12.7742079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100139
2022-12-01T11:12:12.7742456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7742635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7743013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7743187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7743551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7743722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7744095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7744338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7744588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7744836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7745241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7745639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7745852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7746079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7746339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppc71k7fb
2022-12-01T11:12:12.7746618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppc71k7fb/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7746874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph4by_c3p
2022-12-01T11:12:12.7747139Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph4by_c3p/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7747479Z STAGE:2022-12-01 11:03:20 100138:100138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7747816Z STAGE:2022-12-01 11:03:20 100139:100139 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7748441Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.7748537Z   warnings.warn(
2022-12-01T11:12:12.7749205Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2387: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2022-12-01T11:12:12.7749327Z   warnings.warn(
2022-12-01T11:12:12.7749787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7750029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:12:12.7750382Z STAGE:2022-12-01 11:03:20 100138:100138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7750729Z STAGE:2022-12-01 11:03:20 100139:100139 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7751063Z STAGE:2022-12-01 11:03:20 100138:100138 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7751380Z STAGE:2022-12-01 11:03:20 100138:100138 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7751489Z ok (6.866s)
2022-12-01T11:12:12.7751509Z 
2022-12-01T11:12:12.7751777Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7751890Z Ran 1 test in 6.866s
2022-12-01T11:12:12.7751909Z 
2022-12-01T11:12:12.7752002Z OK
2022-12-01T11:12:12.7752021Z 
2022-12-01T11:12:12.7752145Z Generating XML reports...
2022-12-01T11:12:12.7752597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110314.xml
2022-12-01T11:12:12.7752969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7753148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7753507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7753699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7753791Z 
2022-12-01T11:12:12.7753907Z Running tests...
2022-12-01T11:12:12.7754173Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7754489Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7754754Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7754978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100535
2022-12-01T11:12:12.7755197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100536
2022-12-01T11:12:12.7755554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7755730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7756110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7756307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7756677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7756854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7757233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7757422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7757669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7757898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7758299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7758764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7759005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7759234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7759492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0iuw8_wz
2022-12-01T11:12:12.7759763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0iuw8_wz/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7760019Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp70imnj1b
2022-12-01T11:12:12.7760289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp70imnj1b/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7760374Z ok (6.608s)
2022-12-01T11:12:12.7760394Z 
2022-12-01T11:12:12.7760666Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7760783Z Ran 1 test in 6.608s
2022-12-01T11:12:12.7760802Z 
2022-12-01T11:12:12.7760897Z OK
2022-12-01T11:12:12.7760917Z 
2022-12-01T11:12:12.7761040Z Generating XML reports...
2022-12-01T11:12:12.7761497Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110323.xml
2022-12-01T11:12:12.7761872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7762049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7762413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7762606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7762626Z 
2022-12-01T11:12:12.7762736Z Running tests...
2022-12-01T11:12:12.7763000Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7763377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7763652Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7763872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100876
2022-12-01T11:12:12.7764092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100877
2022-12-01T11:12:12.7764462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7764620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7764997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7765190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7765568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7765744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7766125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7766313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7766559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7766785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7767188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7767589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7767866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7768100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7768364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp04yr756
2022-12-01T11:12:12.7768635Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp04yr756/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7768893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpat04475s
2022-12-01T11:12:12.7769160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpat04475s/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7769245Z ok (6.796s)
2022-12-01T11:12:12.7769265Z 
2022-12-01T11:12:12.7769589Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7769703Z Ran 1 test in 6.796s
2022-12-01T11:12:12.7769723Z 
2022-12-01T11:12:12.7769815Z OK
2022-12-01T11:12:12.7769837Z 
2022-12-01T11:12:12.7769960Z Generating XML reports...
2022-12-01T11:12:12.7770417Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110332.xml
2022-12-01T11:12:12.7770793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7770971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7771352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7771529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7771548Z 
2022-12-01T11:12:12.7771659Z Running tests...
2022-12-01T11:12:12.7771922Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7772236Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7772567Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7773321Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.727s)
2022-12-01T11:12:12.7773343Z 
2022-12-01T11:12:12.7773604Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7773717Z Ran 1 test in 1.727s
2022-12-01T11:12:12.7773736Z 
2022-12-01T11:12:12.7773842Z OK (skipped=1)
2022-12-01T11:12:12.7773861Z 
2022-12-01T11:12:12.7773966Z Generating XML reports...
2022-12-01T11:12:12.7774419Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110341.xml
2022-12-01T11:12:12.7774792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7774976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7775356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7775548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7775568Z 
2022-12-01T11:12:12.7775675Z Running tests...
2022-12-01T11:12:12.7775943Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7776258Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7776503Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7777291Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.720s)
2022-12-01T11:12:12.7777317Z 
2022-12-01T11:12:12.7777591Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7777705Z Ran 1 test in 1.720s
2022-12-01T11:12:12.7777724Z 
2022-12-01T11:12:12.7777834Z OK (skipped=1)
2022-12-01T11:12:12.7777853Z 
2022-12-01T11:12:12.7777978Z Generating XML reports...
2022-12-01T11:12:12.7778427Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110345.xml
2022-12-01T11:12:12.7778800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7778977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7779359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7779541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7779561Z 
2022-12-01T11:12:12.7779668Z Running tests...
2022-12-01T11:12:12.7779933Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7780247Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7780540Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7781284Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.777s)
2022-12-01T11:12:12.7781305Z 
2022-12-01T11:12:12.7781571Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7781753Z Ran 1 test in 1.777s
2022-12-01T11:12:12.7781772Z 
2022-12-01T11:12:12.7781885Z OK (skipped=1)
2022-12-01T11:12:12.7781903Z 
2022-12-01T11:12:12.7782011Z Generating XML reports...
2022-12-01T11:12:12.7782468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110349.xml
2022-12-01T11:12:12.7782842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7783019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7783398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7783589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7783609Z 
2022-12-01T11:12:12.7783715Z Running tests...
2022-12-01T11:12:12.7783979Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7784302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7784583Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7784806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101788
2022-12-01T11:12:12.7785028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101789
2022-12-01T11:12:12.7785398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7785577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7785960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7786154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7786574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7786739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7787119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7787308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7787555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7787802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7788203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7788602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7788841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7789073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7789312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltg_5klk
2022-12-01T11:12:12.7789813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltg_5klk/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7790077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpff4v54a8
2022-12-01T11:12:12.7790347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpff4v54a8/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7790450Z ok (7.384s)
2022-12-01T11:12:12.7790471Z 
2022-12-01T11:12:12.7790750Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7790864Z Ran 1 test in 7.384s
2022-12-01T11:12:12.7790960Z 
2022-12-01T11:12:12.7791058Z OK
2022-12-01T11:12:12.7791077Z 
2022-12-01T11:12:12.7791200Z Generating XML reports...
2022-12-01T11:12:12.7791649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110353.xml
2022-12-01T11:12:12.7792023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7792202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7792583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7792774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7792794Z 
2022-12-01T11:12:12.7792902Z Running tests...
2022-12-01T11:12:12.7793165Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7793482Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7793735Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7793958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102111
2022-12-01T11:12:12.7794174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102112
2022-12-01T11:12:12.7794549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7794725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7795106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7795295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7795660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7795838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7796255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7796454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7796700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7796945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7797347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7797743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7797972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7798221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7798447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7798674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7799076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7799471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7799574Z ok (3.944s)
2022-12-01T11:12:12.7799594Z 
2022-12-01T11:12:12.7799859Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7799972Z Ran 1 test in 3.944s
2022-12-01T11:12:12.7799992Z 
2022-12-01T11:12:12.7800084Z OK
2022-12-01T11:12:12.7800103Z 
2022-12-01T11:12:12.7800281Z Generating XML reports...
2022-12-01T11:12:12.7800725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110403.xml
2022-12-01T11:12:12.7801098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7801272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7801653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7801848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7801867Z 
2022-12-01T11:12:12.7801976Z Running tests...
2022-12-01T11:12:12.7802241Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7802558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7802811Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7803019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102296
2022-12-01T11:12:12.7803238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102297
2022-12-01T11:12:12.7803608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7803784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7804165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7804355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7804721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7804894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7805310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7805511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7805759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7806004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7806408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7806806Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7807038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7807280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7807516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7807741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7808138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7808530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7808633Z ok (4.204s)
2022-12-01T11:12:12.7808652Z 
2022-12-01T11:12:12.7808915Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7809028Z Ran 1 test in 4.204s
2022-12-01T11:12:12.7809048Z 
2022-12-01T11:12:12.7809139Z OK
2022-12-01T11:12:12.7809158Z 
2022-12-01T11:12:12.7809282Z Generating XML reports...
2022-12-01T11:12:12.7809731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110409.xml
2022-12-01T11:12:12.7810143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7810320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7810704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7810896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7810915Z 
2022-12-01T11:12:12.7811023Z Running tests...
2022-12-01T11:12:12.7811290Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7811603Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7811881Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7812635Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.752s)
2022-12-01T11:12:12.7812657Z 
2022-12-01T11:12:12.7812918Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7813012Z Ran 1 test in 1.752s
2022-12-01T11:12:12.7813031Z 
2022-12-01T11:12:12.7813139Z OK (skipped=1)
2022-12-01T11:12:12.7813158Z 
2022-12-01T11:12:12.7813281Z Generating XML reports...
2022-12-01T11:12:12.7813729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110415.xml
2022-12-01T11:12:12.7814102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7814281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7814712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7814913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7814932Z 
2022-12-01T11:12:12.7815022Z Running tests...
2022-12-01T11:12:12.7815288Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7815606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7815882Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7816625Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.857s)
2022-12-01T11:12:12.7816649Z 
2022-12-01T11:12:12.7816914Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7817032Z Ran 1 test in 1.857s
2022-12-01T11:12:12.7817051Z 
2022-12-01T11:12:12.7817157Z OK (skipped=1)
2022-12-01T11:12:12.7817176Z 
2022-12-01T11:12:12.7817300Z Generating XML reports...
2022-12-01T11:12:12.7817748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110419.xml
2022-12-01T11:12:12.7818105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7818281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7818664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7818854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7818918Z 
2022-12-01T11:12:12.7819031Z Running tests...
2022-12-01T11:12:12.7819302Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7819618Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7819895Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7820099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102860
2022-12-01T11:12:12.7820317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102861
2022-12-01T11:12:12.7820691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7820867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7821248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7821446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7821812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7821985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7822357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7822530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7822777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7823024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7823425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7823870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7824108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7824338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7824441Z ok (4.058s)
2022-12-01T11:12:12.7824460Z 
2022-12-01T11:12:12.7824727Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7824822Z Ran 1 test in 4.058s
2022-12-01T11:12:12.7824841Z 
2022-12-01T11:12:12.7824933Z OK
2022-12-01T11:12:12.7824952Z 
2022-12-01T11:12:12.7825076Z Generating XML reports...
2022-12-01T11:12:12.7825530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110423.xml
2022-12-01T11:12:12.7825898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7826083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7826469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7826660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7826681Z 
2022-12-01T11:12:12.7826770Z Running tests...
2022-12-01T11:12:12.7827034Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7827351Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7827604Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7827623Z 
2022-12-01T11:12:12.7827884Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7827996Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7828016Z 
2022-12-01T11:12:12.7828175Z OK (skipped=1)
2022-12-01T11:12:12.7828194Z 
2022-12-01T11:12:12.7828322Z Generating XML reports...
2022-12-01T11:12:12.7828774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110429.xml
2022-12-01T11:12:12.7829129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7829306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7829931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7830124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7830144Z 
2022-12-01T11:12:12.7830252Z Running tests...
2022-12-01T11:12:12.7830518Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7830837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7831106Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7831126Z 
2022-12-01T11:12:12.7831388Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7831483Z Ran 1 test in 0.003s
2022-12-01T11:12:12.7831502Z 
2022-12-01T11:12:12.7831610Z OK (skipped=1)
2022-12-01T11:12:12.7831629Z 
2022-12-01T11:12:12.7831752Z Generating XML reports...
2022-12-01T11:12:12.7832206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110431.xml
2022-12-01T11:12:12.7832570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7844223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7844722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7845080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7845106Z 
2022-12-01T11:12:12.7845230Z Running tests...
2022-12-01T11:12:12.7845522Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7845863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7846126Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7846343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103295
2022-12-01T11:12:12.7846577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103296
2022-12-01T11:12:12.7846977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7847164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7847583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7847790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7848185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7848370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7848757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7848958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7849215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7849473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7849902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7850419Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7850664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7850907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7851265Z STAGE:2022-12-01 11:04:39 103296:103296 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7851596Z STAGE:2022-12-01 11:04:39 103295:103295 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7851956Z STAGE:2022-12-01 11:04:40 103295:103295 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7852319Z STAGE:2022-12-01 11:04:40 103296:103296 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7852680Z STAGE:2022-12-01 11:04:41 103295:103295 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7853032Z STAGE:2022-12-01 11:04:41 103296:103296 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.7853391Z STAGE:2022-12-01 11:04:41 103296:103296 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7853748Z STAGE:2022-12-01 11:04:41 103295:103295 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.7853850Z ok (7.750s)
2022-12-01T11:12:12.7853871Z 
2022-12-01T11:12:12.7854159Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7854258Z Ran 1 test in 7.751s
2022-12-01T11:12:12.7854278Z 
2022-12-01T11:12:12.7854371Z OK
2022-12-01T11:12:12.7854391Z 
2022-12-01T11:12:12.7854520Z Generating XML reports...
2022-12-01T11:12:12.7855006Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110434.xml
2022-12-01T11:12:12.7855459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7855654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7856061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7856264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7856286Z 
2022-12-01T11:12:12.7856397Z Running tests...
2022-12-01T11:12:12.7856665Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7857002Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7857282Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7857302Z 
2022-12-01T11:12:12.7857586Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7857707Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7857727Z 
2022-12-01T11:12:12.7857839Z OK (skipped=1)
2022-12-01T11:12:12.7857859Z 
2022-12-01T11:12:12.7857989Z Generating XML reports...
2022-12-01T11:12:12.7858468Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110444.xml
2022-12-01T11:12:12.7858846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7859031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7859442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7859643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7859663Z 
2022-12-01T11:12:12.7859773Z Running tests...
2022-12-01T11:12:12.7860118Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7860460Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7860734Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.7860754Z 
2022-12-01T11:12:12.7861033Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7861131Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7861150Z 
2022-12-01T11:12:12.7861264Z OK (skipped=1)
2022-12-01T11:12:12.7861283Z 
2022-12-01T11:12:12.7861411Z Generating XML reports...
2022-12-01T11:12:12.7861891Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110446.xml
2022-12-01T11:12:12.7862288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7862474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7862887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7863091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7863110Z 
2022-12-01T11:12:12.7863220Z Running tests...
2022-12-01T11:12:12.7863484Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7863817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7864078Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7864317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103788
2022-12-01T11:12:12.7864552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103789
2022-12-01T11:12:12.7864952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7865193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7865613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7865796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7866189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7866373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7866776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7866975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7867233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7867497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7867921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7868343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7868568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7868809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7868913Z ok (6.959s)
2022-12-01T11:12:12.7868933Z 
2022-12-01T11:12:12.7869217Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7869334Z Ran 1 test in 6.959s
2022-12-01T11:12:12.7869746Z 
2022-12-01T11:12:12.7869858Z OK
2022-12-01T11:12:12.7869878Z 
2022-12-01T11:12:12.7870015Z Generating XML reports...
2022-12-01T11:12:12.7870597Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110448.xml
2022-12-01T11:12:12.7870960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7871141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7871523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7871716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7871736Z 
2022-12-01T11:12:12.7871846Z Running tests...
2022-12-01T11:12:12.7872113Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7872431Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7872704Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7873472Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2022-12-01T11:12:12.7873493Z 
2022-12-01T11:12:12.7873763Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7873860Z Ran 1 test in 1.677s
2022-12-01T11:12:12.7873879Z 
2022-12-01T11:12:12.7873987Z OK (skipped=1)
2022-12-01T11:12:12.7874006Z 
2022-12-01T11:12:12.7874129Z Generating XML reports...
2022-12-01T11:12:12.7874582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110457.xml
2022-12-01T11:12:12.7874960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7875140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7875585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7875789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7875809Z 
2022-12-01T11:12:12.7875917Z Running tests...
2022-12-01T11:12:12.7876167Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7876485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7876734Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7876957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104288
2022-12-01T11:12:12.7877178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104289
2022-12-01T11:12:12.7877567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7877746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7878132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7878308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7878679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7878857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7879233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7879424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7879675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7879980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7880386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7880788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7881003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7881247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7881475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7881716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7882119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7882516Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7882622Z ok (4.071s)
2022-12-01T11:12:12.7882642Z 
2022-12-01T11:12:12.7882911Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7883007Z Ran 1 test in 4.071s
2022-12-01T11:12:12.7883044Z 
2022-12-01T11:12:12.7883118Z OK
2022-12-01T11:12:12.7883136Z 
2022-12-01T11:12:12.7883262Z Generating XML reports...
2022-12-01T11:12:12.7883719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110501.xml
2022-12-01T11:12:12.7884094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7884271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7884754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7884957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7884977Z 
2022-12-01T11:12:12.7885087Z Running tests...
2022-12-01T11:12:12.7885337Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7885652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7885900Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7886124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104604
2022-12-01T11:12:12.7886342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104605
2022-12-01T11:12:12.7886711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7886895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7887277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7887451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7887818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7887994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7888379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7888569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7888816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7889061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7889529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7889931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7890144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7890375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7890479Z ok (6.881s)
2022-12-01T11:12:12.7890499Z 
2022-12-01T11:12:12.7890766Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7890879Z Ran 1 test in 6.881s
2022-12-01T11:12:12.7890898Z 
2022-12-01T11:12:12.7890990Z OK
2022-12-01T11:12:12.7891008Z 
2022-12-01T11:12:12.7891133Z Generating XML reports...
2022-12-01T11:12:12.7891595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110507.xml
2022-12-01T11:12:12.7891969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7892128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7892513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7892708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7892728Z 
2022-12-01T11:12:12.7892838Z Running tests...
2022-12-01T11:12:12.7893100Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7893415Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7893661Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7893932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104844
2022-12-01T11:12:12.7894142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104845
2022-12-01T11:12:12.7894519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7894696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7895078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7895271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7895638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7895813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7896188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7896385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7896616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7896861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7897268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7897667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7897897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7898124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7898277Z ok (4.206s)
2022-12-01T11:12:12.7898296Z 
2022-12-01T11:12:12.7898566Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7898662Z Ran 1 test in 4.206s
2022-12-01T11:12:12.7898701Z 
2022-12-01T11:12:12.7898775Z OK
2022-12-01T11:12:12.7898794Z 
2022-12-01T11:12:12.7898921Z Generating XML reports...
2022-12-01T11:12:12.7899381Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110516.xml
2022-12-01T11:12:12.7899752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7899929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7900306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7900499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7900518Z 
2022-12-01T11:12:12.7900633Z Running tests...
2022-12-01T11:12:12.7900882Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7901199Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7901469Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7901696Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105164
2022-12-01T11:12:12.7901916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105165
2022-12-01T11:12:12.7902291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7902468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7902845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7903022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7903429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7903612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7903987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7904177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7904423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7904669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7905069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7905470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7905692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7905922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7906164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7906407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7906805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7907201Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7907304Z ok (3.846s)
2022-12-01T11:12:12.7907324Z 
2022-12-01T11:12:12.7907589Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7907754Z Ran 1 test in 3.846s
2022-12-01T11:12:12.7907777Z 
2022-12-01T11:12:12.7907851Z OK
2022-12-01T11:12:12.7907870Z 
2022-12-01T11:12:12.7907994Z Generating XML reports...
2022-12-01T11:12:12.7908447Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110523.xml
2022-12-01T11:12:12.7908817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7908995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7909579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7909781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7909803Z 
2022-12-01T11:12:12.7909912Z Running tests...
2022-12-01T11:12:12.7910183Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7910491Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7910753Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7910976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105473
2022-12-01T11:12:12.7911198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105474
2022-12-01T11:12:12.7911637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7911814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7912195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7912390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7912817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7913002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7913379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7913570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7913815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7914060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7914463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7914860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7915104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7915329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7915555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7915790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7916188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7916579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7916684Z ok (4.087s)
2022-12-01T11:12:12.7916703Z 
2022-12-01T11:12:12.7916967Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7917150Z Ran 1 test in 4.087s
2022-12-01T11:12:12.7917169Z 
2022-12-01T11:12:12.7917262Z OK
2022-12-01T11:12:12.7917281Z 
2022-12-01T11:12:12.7917392Z Generating XML reports...
2022-12-01T11:12:12.7917852Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110529.xml
2022-12-01T11:12:12.7918227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7918404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7918786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7918978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7918999Z 
2022-12-01T11:12:12.7919107Z Running tests...
2022-12-01T11:12:12.7919370Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7919674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7919942Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7920165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105798
2022-12-01T11:12:12.7920384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105799
2022-12-01T11:12:12.7920754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7920929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7921312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7921506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7921875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7922090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7922484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7922675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7922920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7923166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7923568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7923964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7924197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7924414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7924676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwe5evvdp
2022-12-01T11:12:12.7924952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwe5evvdp/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7925207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg481heid
2022-12-01T11:12:12.7925478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg481heid/_remote_module_non_scriptable.py
2022-12-01T11:12:12.7925582Z ok (6.345s)
2022-12-01T11:12:12.7925601Z 
2022-12-01T11:12:12.7925873Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7925988Z Ran 1 test in 6.345s
2022-12-01T11:12:12.7926008Z 
2022-12-01T11:12:12.7926102Z OK
2022-12-01T11:12:12.7926121Z 
2022-12-01T11:12:12.7926227Z Generating XML reports...
2022-12-01T11:12:12.7926747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110535.xml
2022-12-01T11:12:12.7927124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7927300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7927679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7927870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7927890Z 
2022-12-01T11:12:12.7927997Z Running tests...
2022-12-01T11:12:12.7928259Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7928557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7928796Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2022-12-01T11:12:12.7928820Z 
2022-12-01T11:12:12.7929083Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7929194Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7929213Z 
2022-12-01T11:12:12.7929320Z OK (skipped=1)
2022-12-01T11:12:12.7929339Z 
2022-12-01T11:12:12.7929462Z Generating XML reports...
2022-12-01T11:12:12.7929914Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110544.xml
2022-12-01T11:12:12.7930284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7930460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7930823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7931015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7931038Z 
2022-12-01T11:12:12.7931149Z Running tests...
2022-12-01T11:12:12.7931458Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7931786Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7932023Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-12-01T11:12:12.7932042Z 
2022-12-01T11:12:12.7932303Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7932415Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7932434Z 
2022-12-01T11:12:12.7932543Z OK (skipped=1)
2022-12-01T11:12:12.7932561Z 
2022-12-01T11:12:12.7932668Z Generating XML reports...
2022-12-01T11:12:12.7933122Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110546.xml
2022-12-01T11:12:12.7933496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7933679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7934062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7934254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7934275Z 
2022-12-01T11:12:12.7934383Z Running tests...
2022-12-01T11:12:12.7934649Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7934948Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7935218Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-12-01T11:12:12.7935237Z 
2022-12-01T11:12:12.7935497Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7935609Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7935676Z 
2022-12-01T11:12:12.7935788Z OK (skipped=1)
2022-12-01T11:12:12.7935807Z 
2022-12-01T11:12:12.7935934Z Generating XML reports...
2022-12-01T11:12:12.7936388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110548.xml
2022-12-01T11:12:12.7936762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7936940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7937305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7937501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7937520Z 
2022-12-01T11:12:12.7937628Z Running tests...
2022-12-01T11:12:12.7937893Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7938208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7938477Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2022-12-01T11:12:12.7938497Z 
2022-12-01T11:12:12.7938758Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7938871Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7938891Z 
2022-12-01T11:12:12.7938998Z OK (skipped=1)
2022-12-01T11:12:12.7939017Z 
2022-12-01T11:12:12.7939124Z Generating XML reports...
2022-12-01T11:12:12.7939577Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110550.xml
2022-12-01T11:12:12.7939953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7940129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7940510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7940751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7940772Z 
2022-12-01T11:12:12.7940884Z Running tests...
2022-12-01T11:12:12.7941151Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7941466Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7941732Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7941954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106500
2022-12-01T11:12:12.7942174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106501
2022-12-01T11:12:12.7942546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7942722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7943109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7943300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7943665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7943820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7944201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7944394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7944639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7944886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7945366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7945762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7945994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7946238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7946445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7946686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7947088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7947482Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7947732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.7948128Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7948371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.7948763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7948997Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:12:12.7949080Z ok (22.695s)
2022-12-01T11:12:12.7949100Z 
2022-12-01T11:12:12.7949600Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7949727Z Ran 1 test in 22.695s
2022-12-01T11:12:12.7949751Z 
2022-12-01T11:12:12.7949848Z OK
2022-12-01T11:12:12.7949867Z 
2022-12-01T11:12:12.7950077Z Generating XML reports...
2022-12-01T11:12:12.7950553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110552.xml
2022-12-01T11:12:12.7950924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7951104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7951487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7951662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7951682Z 
2022-12-01T11:12:12.7951790Z Running tests...
2022-12-01T11:12:12.7952054Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7952370Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7952680Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7952901Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107172
2022-12-01T11:12:12.7953122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107173
2022-12-01T11:12:12.7953492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7953650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7954041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7954232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7954601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7954849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7955231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7955422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7955669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7955911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7956295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7956693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7956925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7957177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.7957402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7957638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.7958034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7958425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.7958665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.7958889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.7959333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7959744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.7959982Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:12:12.7960213Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:12:12.7960315Z ok (22.700s)
2022-12-01T11:12:12.7960335Z 
2022-12-01T11:12:12.7960601Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7960715Z Ran 1 test in 22.700s
2022-12-01T11:12:12.7960736Z 
2022-12-01T11:12:12.7960832Z OK
2022-12-01T11:12:12.7960852Z 
2022-12-01T11:12:12.7960958Z Generating XML reports...
2022-12-01T11:12:12.7961414Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110617.xml
2022-12-01T11:12:12.7961798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7961975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7962355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7962549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7962568Z 
2022-12-01T11:12:12.7962676Z Running tests...
2022-12-01T11:12:12.7962939Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7963236Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7963653Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7963674Z 
2022-12-01T11:12:12.7963993Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7964112Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7964131Z 
2022-12-01T11:12:12.7964245Z OK (skipped=1)
2022-12-01T11:12:12.7964263Z 
2022-12-01T11:12:12.7964387Z Generating XML reports...
2022-12-01T11:12:12.7964840Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110642.xml
2022-12-01T11:12:12.7965215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7965392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7965757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7965950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7965970Z 
2022-12-01T11:12:12.7966079Z Running tests...
2022-12-01T11:12:12.7966351Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7966670Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7967064Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7967084Z 
2022-12-01T11:12:12.7967341Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7967453Z Ran 1 test in 0.003s
2022-12-01T11:12:12.7967472Z 
2022-12-01T11:12:12.7967580Z OK (skipped=1)
2022-12-01T11:12:12.7967599Z 
2022-12-01T11:12:12.7967704Z Generating XML reports...
2022-12-01T11:12:12.7968153Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110644.xml
2022-12-01T11:12:12.7968526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7968707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7969135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7969335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7969354Z 
2022-12-01T11:12:12.7969507Z Running tests...
2022-12-01T11:12:12.7969776Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7970069Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7970493Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7970513Z 
2022-12-01T11:12:12.7970771Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7970884Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7970904Z 
2022-12-01T11:12:12.7971011Z OK (skipped=1)
2022-12-01T11:12:12.7971034Z 
2022-12-01T11:12:12.7971160Z Generating XML reports...
2022-12-01T11:12:12.7971616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110646.xml
2022-12-01T11:12:12.7971989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7972167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7972552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7972727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7972747Z 
2022-12-01T11:12:12.7972856Z Running tests...
2022-12-01T11:12:12.7973123Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7973437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7973915Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7973936Z 
2022-12-01T11:12:12.7974197Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7974313Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7974332Z 
2022-12-01T11:12:12.7974441Z OK (skipped=1)
2022-12-01T11:12:12.7974459Z 
2022-12-01T11:12:12.7974565Z Generating XML reports...
2022-12-01T11:12:12.7975015Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110649.xml
2022-12-01T11:12:12.7975384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7975561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7975940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7976140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7976159Z 
2022-12-01T11:12:12.7976265Z Running tests...
2022-12-01T11:12:12.7976529Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7976842Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7977236Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.7977274Z 
2022-12-01T11:12:12.7977522Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7977634Z Ran 1 test in 0.002s
2022-12-01T11:12:12.7977653Z 
2022-12-01T11:12:12.7977761Z OK (skipped=1)
2022-12-01T11:12:12.7977780Z 
2022-12-01T11:12:12.7977904Z Generating XML reports...
2022-12-01T11:12:12.7978358Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110651.xml
2022-12-01T11:12:12.7978799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7978990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7979377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7979551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7979570Z 
2022-12-01T11:12:12.7979678Z Running tests...
2022-12-01T11:12:12.7979946Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7980260Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7980537Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7980763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108531
2022-12-01T11:12:12.7980985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108532
2022-12-01T11:12:12.7981360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7981519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7981901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7982095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7982463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7982635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7983015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7983264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7983516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7983765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7984151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7984555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7984788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7985018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7985121Z ok (5.587s)
2022-12-01T11:12:12.7985145Z 
2022-12-01T11:12:12.7985415Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7985532Z Ran 1 test in 5.587s
2022-12-01T11:12:12.7985552Z 
2022-12-01T11:12:12.7985646Z OK
2022-12-01T11:12:12.7985665Z 
2022-12-01T11:12:12.7985792Z Generating XML reports...
2022-12-01T11:12:12.7986228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110653.xml
2022-12-01T11:12:12.7986603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7986777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7987156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7987348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7987368Z 
2022-12-01T11:12:12.7987478Z Running tests...
2022-12-01T11:12:12.7987747Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7988108Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7988375Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7988598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108896
2022-12-01T11:12:12.7988821Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108897
2022-12-01T11:12:12.7989196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7989593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7989993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7990189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7990568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7990746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7991107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7991298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7991545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7991791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.7992197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7992602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.7992920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.7993150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.7993239Z ok (5.632s)
2022-12-01T11:12:12.7993277Z 
2022-12-01T11:12:12.7993530Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7993643Z Ran 1 test in 5.633s
2022-12-01T11:12:12.7993663Z 
2022-12-01T11:12:12.7993756Z OK
2022-12-01T11:12:12.7993775Z 
2022-12-01T11:12:12.7993899Z Generating XML reports...
2022-12-01T11:12:12.7994354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110701.xml
2022-12-01T11:12:12.7994727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7994903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7995291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7995468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7995488Z 
2022-12-01T11:12:12.7995597Z Running tests...
2022-12-01T11:12:12.7995865Z ----------------------------------------------------------------------
2022-12-01T11:12:12.7996179Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.7996453Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.7996674Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109240
2022-12-01T11:12:12.7996893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109242
2022-12-01T11:12:12.7997266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7997487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7997884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7998075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7998437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.7998610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.7998982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.7999170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.7999418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.7999668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8000053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8000448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8000680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8000910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8001012Z ok (6.751s)
2022-12-01T11:12:12.8001031Z 
2022-12-01T11:12:12.8001297Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8001411Z Ran 1 test in 6.751s
2022-12-01T11:12:12.8001431Z 
2022-12-01T11:12:12.8001524Z OK
2022-12-01T11:12:12.8001543Z 
2022-12-01T11:12:12.8001721Z Generating XML reports...
2022-12-01T11:12:12.8002164Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110708.xml
2022-12-01T11:12:12.8002538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8002712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8003094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8003286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8003305Z 
2022-12-01T11:12:12.8003413Z Running tests...
2022-12-01T11:12:12.8003676Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8003995Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8004246Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8004476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109644
2022-12-01T11:12:12.8004694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109645
2022-12-01T11:12:12.8005065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8005241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8005626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8005819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8006183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8006355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8006765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8006966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8007213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8007461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8007864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8008261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8008492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8008725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8008813Z ok (5.599s)
2022-12-01T11:12:12.8008851Z 
2022-12-01T11:12:12.8009102Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8009217Z Ran 1 test in 5.600s
2022-12-01T11:12:12.8009237Z 
2022-12-01T11:12:12.8009329Z OK
2022-12-01T11:12:12.8009348Z 
2022-12-01T11:12:12.8009470Z Generating XML reports...
2022-12-01T11:12:12.8009927Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110717.xml
2022-12-01T11:12:12.8010300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8010478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8010860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8011036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8011103Z 
2022-12-01T11:12:12.8011221Z Running tests...
2022-12-01T11:12:12.8011489Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8011803Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8012075Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8012830Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.692s)
2022-12-01T11:12:12.8012851Z 
2022-12-01T11:12:12.8013113Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8013226Z Ran 1 test in 1.692s
2022-12-01T11:12:12.8013246Z 
2022-12-01T11:12:12.8013359Z OK (skipped=1)
2022-12-01T11:12:12.8013378Z 
2022-12-01T11:12:12.8013502Z Generating XML reports...
2022-12-01T11:12:12.8013940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110725.xml
2022-12-01T11:12:12.8014312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8014489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8014870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8015063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8015083Z 
2022-12-01T11:12:12.8015192Z Running tests...
2022-12-01T11:12:12.8015458Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8015776Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8016071Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:12:12.8016111Z 
2022-12-01T11:12:12.8016365Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8016479Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8016498Z 
2022-12-01T11:12:12.8016607Z OK (skipped=1)
2022-12-01T11:12:12.8016626Z 
2022-12-01T11:12:12.8016749Z Generating XML reports...
2022-12-01T11:12:12.8017198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110729.xml
2022-12-01T11:12:12.8017574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8017749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8018131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8018319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8018338Z 
2022-12-01T11:12:12.8018448Z Running tests...
2022-12-01T11:12:12.8018712Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8019028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8019298Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:12:12.8019317Z 
2022-12-01T11:12:12.8019578Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8019691Z Ran 1 test in 0.003s
2022-12-01T11:12:12.8019711Z 
2022-12-01T11:12:12.8019818Z OK (skipped=1)
2022-12-01T11:12:12.8019837Z 
2022-12-01T11:12:12.8019961Z Generating XML reports...
2022-12-01T11:12:12.8020394Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110731.xml
2022-12-01T11:12:12.8020828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8021008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8021393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8021585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8021604Z 
2022-12-01T11:12:12.8021714Z Running tests...
2022-12-01T11:12:12.8021978Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8022292Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8023509Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:12:12.8023534Z 
2022-12-01T11:12:12.8023788Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8023907Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8023927Z 
2022-12-01T11:12:12.8024035Z OK (skipped=1)
2022-12-01T11:12:12.8024054Z 
2022-12-01T11:12:12.8024179Z Generating XML reports...
2022-12-01T11:12:12.8024633Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110733.xml
2022-12-01T11:12:12.8025008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8025185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8025571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8025768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8025788Z 
2022-12-01T11:12:12.8025883Z Running tests...
2022-12-01T11:12:12.8026204Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8026530Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8026833Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8027056Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110403
2022-12-01T11:12:12.8027276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110404
2022-12-01T11:12:12.8027652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8027830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8028196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8028395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8028766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8028942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8029315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8029688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8029940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8030188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8030599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8030990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8031344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8031572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8031678Z ok (3.946s)
2022-12-01T11:12:12.8031698Z 
2022-12-01T11:12:12.8031966Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8032079Z Ran 1 test in 3.946s
2022-12-01T11:12:12.8032098Z 
2022-12-01T11:12:12.8032193Z OK
2022-12-01T11:12:12.8032213Z 
2022-12-01T11:12:12.8032338Z Generating XML reports...
2022-12-01T11:12:12.8032774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110735.xml
2022-12-01T11:12:12.8033149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8033331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8033718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8033914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8033933Z 
2022-12-01T11:12:12.8034041Z Running tests...
2022-12-01T11:12:12.8034304Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8034621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8034917Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8035120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110657
2022-12-01T11:12:12.8035342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110658
2022-12-01T11:12:12.8035780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8035968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8036353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8036546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8036916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8037092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8037470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8037646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8037893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8038148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8038551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8038951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8039184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8039415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8039518Z ok (4.067s)
2022-12-01T11:12:12.8039538Z 
2022-12-01T11:12:12.8039785Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8039900Z Ran 1 test in 4.067s
2022-12-01T11:12:12.8039919Z 
2022-12-01T11:12:12.8040064Z OK
2022-12-01T11:12:12.8040083Z 
2022-12-01T11:12:12.8040209Z Generating XML reports...
2022-12-01T11:12:12.8040666Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110741.xml
2022-12-01T11:12:12.8041038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8041215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8041597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8041791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8041810Z 
2022-12-01T11:12:12.8041901Z Running tests...
2022-12-01T11:12:12.8042170Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8042485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8042772Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:12:12.8042791Z 
2022-12-01T11:12:12.8043055Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8043170Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8043189Z 
2022-12-01T11:12:12.8043297Z OK (skipped=1)
2022-12-01T11:12:12.8043316Z 
2022-12-01T11:12:12.8043442Z Generating XML reports...
2022-12-01T11:12:12.8043896Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110748.xml
2022-12-01T11:12:12.8044250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8044429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8044811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8045054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8045075Z 
2022-12-01T11:12:12.8045188Z Running tests...
2022-12-01T11:12:12.8045453Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8045769Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8046071Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:12:12.8046091Z 
2022-12-01T11:12:12.8046355Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8046449Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8046468Z 
2022-12-01T11:12:12.8046578Z OK (skipped=1)
2022-12-01T11:12:12.8046598Z 
2022-12-01T11:12:12.8046722Z Generating XML reports...
2022-12-01T11:12:12.8047174Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110750.xml
2022-12-01T11:12:12.8047554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8047731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8048110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8048303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8048322Z 
2022-12-01T11:12:12.8048412Z Running tests...
2022-12-01T11:12:12.8048685Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8048998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8049278Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8050087Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.682s)
2022-12-01T11:12:12.8050108Z 
2022-12-01T11:12:12.8050371Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8050484Z Ran 1 test in 1.682s
2022-12-01T11:12:12.8050503Z 
2022-12-01T11:12:12.8050610Z OK (skipped=1)
2022-12-01T11:12:12.8050629Z 
2022-12-01T11:12:12.8050755Z Generating XML reports...
2022-12-01T11:12:12.8051206Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110752.xml
2022-12-01T11:12:12.8051561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8051739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8052128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8052321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8052340Z 
2022-12-01T11:12:12.8052450Z Running tests...
2022-12-01T11:12:12.8052716Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8053033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8053313Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8053516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111367
2022-12-01T11:12:12.8053737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111368
2022-12-01T11:12:12.8054113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8054337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8054728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8054923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8055287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8055462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8055836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8056011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8056261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8056517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8056922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8057323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8057555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8057785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8058044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2makhrqf
2022-12-01T11:12:12.8058320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2makhrqf/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8058558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pokkq4m
2022-12-01T11:12:12.8058893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pokkq4m/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8058995Z ok (7.637s)
2022-12-01T11:12:12.8059015Z 
2022-12-01T11:12:12.8059285Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8059398Z Ran 1 test in 7.637s
2022-12-01T11:12:12.8059417Z 
2022-12-01T11:12:12.8059514Z OK
2022-12-01T11:12:12.8059534Z 
2022-12-01T11:12:12.8059658Z Generating XML reports...
2022-12-01T11:12:12.8060112Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110756.xml
2022-12-01T11:12:12.8060467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8060645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8061027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8061229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8061248Z 
2022-12-01T11:12:12.8061359Z Running tests...
2022-12-01T11:12:12.8061623Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8061942Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8062216Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8062437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111765
2022-12-01T11:12:12.8062640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111766
2022-12-01T11:12:12.8063017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8063194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8063626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8063822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8064190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8064366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8064749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8064918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8065165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8065411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8065823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8066225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8066456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8066688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8066793Z ok (7.064s)
2022-12-01T11:12:12.8066813Z 
2022-12-01T11:12:12.8067079Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8067173Z Ran 1 test in 7.064s
2022-12-01T11:12:12.8067192Z 
2022-12-01T11:12:12.8067285Z OK
2022-12-01T11:12:12.8067304Z 
2022-12-01T11:12:12.8067427Z Generating XML reports...
2022-12-01T11:12:12.8067878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110806.xml
2022-12-01T11:12:12.8068311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8068488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8068876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8069070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8069090Z 
2022-12-01T11:12:12.8069203Z Running tests...
2022-12-01T11:12:12.8069709Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8070074Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8070366Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8070591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112140
2022-12-01T11:12:12.8070823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112141
2022-12-01T11:12:12.8071194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8071372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8071756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8071930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8072295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8072470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8072846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8073046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8073371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8073627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8074030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8074432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8074644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8074875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8074978Z ok (6.808s)
2022-12-01T11:12:12.8074998Z 
2022-12-01T11:12:12.8075266Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8075385Z Ran 1 test in 6.808s
2022-12-01T11:12:12.8075407Z 
2022-12-01T11:12:12.8075502Z OK
2022-12-01T11:12:12.8075521Z 
2022-12-01T11:12:12.8075647Z Generating XML reports...
2022-12-01T11:12:12.8076105Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110815.xml
2022-12-01T11:12:12.8076459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8076637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8077017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8077209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8077228Z 
2022-12-01T11:12:12.8077339Z Running tests...
2022-12-01T11:12:12.8077606Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8077999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8078280Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8079035Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.663s)
2022-12-01T11:12:12.8079056Z 
2022-12-01T11:12:12.8079322Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8079418Z Ran 1 test in 1.663s
2022-12-01T11:12:12.8079437Z 
2022-12-01T11:12:12.8079547Z OK (skipped=1)
2022-12-01T11:12:12.8079566Z 
2022-12-01T11:12:12.8079692Z Generating XML reports...
2022-12-01T11:12:12.8080152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110824.xml
2022-12-01T11:12:12.8080526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8080705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8081086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8081281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8081301Z 
2022-12-01T11:12:12.8081414Z Running tests...
2022-12-01T11:12:12.8081660Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8081977Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8082275Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8083085Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.661s)
2022-12-01T11:12:12.8083107Z 
2022-12-01T11:12:12.8083377Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8083491Z Ran 1 test in 1.662s
2022-12-01T11:12:12.8083510Z 
2022-12-01T11:12:12.8083620Z OK (skipped=1)
2022-12-01T11:12:12.8083639Z 
2022-12-01T11:12:12.8083764Z Generating XML reports...
2022-12-01T11:12:12.8084218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110828.xml
2022-12-01T11:12:12.8084573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8084754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8085143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8085337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8085357Z 
2022-12-01T11:12:12.8085467Z Running tests...
2022-12-01T11:12:12.8085734Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8086052Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8086364Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8086590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112837
2022-12-01T11:12:12.8086793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112838
2022-12-01T11:12:12.8087230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8087403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8087785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8087978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8088349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8088525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8088908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8089100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8089329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8089581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8089985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8090379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8090612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8090842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8090996Z skip: Need at least 4 CUDA devices (3.986s)
2022-12-01T11:12:12.8091015Z 
2022-12-01T11:12:12.8091283Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8091377Z Ran 1 test in 3.986s
2022-12-01T11:12:12.8091422Z 
2022-12-01T11:12:12.8091512Z OK (skipped=1)
2022-12-01T11:12:12.8091531Z 
2022-12-01T11:12:12.8091701Z Generating XML reports...
2022-12-01T11:12:12.8092166Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110832.xml
2022-12-01T11:12:12.8092538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8092714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8093095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8093287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8093307Z 
2022-12-01T11:12:12.8093416Z Running tests...
2022-12-01T11:12:12.8093660Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8093974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8094309Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8094531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112979
2022-12-01T11:12:12.8094752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112980
2022-12-01T11:12:12.8095126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8095303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8095686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8095879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8096225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8096459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8096835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8097022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8097270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8097519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8097923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8098322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8098536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8098772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8098926Z skip: Need at least 4 CUDA devices (3.717s)
2022-12-01T11:12:12.8098945Z 
2022-12-01T11:12:12.8099211Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8099326Z Ran 1 test in 3.717s
2022-12-01T11:12:12.8099346Z 
2022-12-01T11:12:12.8099456Z OK (skipped=1)
2022-12-01T11:12:12.8099475Z 
2022-12-01T11:12:12.8099601Z Generating XML reports...
2022-12-01T11:12:12.8100056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110838.xml
2022-12-01T11:12:12.8100427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8100585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8101017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8101215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8101235Z 
2022-12-01T11:12:12.8101344Z Running tests...
2022-12-01T11:12:12.8101612Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8101928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8102216Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8102961Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.591s)
2022-12-01T11:12:12.8102985Z 
2022-12-01T11:12:12.8103253Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8103371Z Ran 1 test in 1.591s
2022-12-01T11:12:12.8103390Z 
2022-12-01T11:12:12.8103480Z OK (skipped=1)
2022-12-01T11:12:12.8103498Z 
2022-12-01T11:12:12.8103626Z Generating XML reports...
2022-12-01T11:12:12.8104077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110844.xml
2022-12-01T11:12:12.8104445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8104627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8105005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8105197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8105217Z 
2022-12-01T11:12:12.8105383Z Running tests...
2022-12-01T11:12:12.8105638Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8105950Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8106221Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8106241Z 
2022-12-01T11:12:12.8106505Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8106617Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8106636Z 
2022-12-01T11:12:12.8106745Z OK (skipped=1)
2022-12-01T11:12:12.8106765Z 
2022-12-01T11:12:12.8106889Z Generating XML reports...
2022-12-01T11:12:12.8107342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110848.xml
2022-12-01T11:12:12.8107718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8107881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8108268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8108461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8108481Z 
2022-12-01T11:12:12.8108591Z Running tests...
2022-12-01T11:12:12.8108856Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8109171Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8109660Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8109682Z 
2022-12-01T11:12:12.8109955Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8110070Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8110090Z 
2022-12-01T11:12:12.8110189Z OK (skipped=1)
2022-12-01T11:12:12.8110208Z 
2022-12-01T11:12:12.8110335Z Generating XML reports...
2022-12-01T11:12:12.8110861Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110850.xml
2022-12-01T11:12:12.8111248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8111424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8111805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8111999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8112018Z 
2022-12-01T11:12:12.8112129Z Running tests...
2022-12-01T11:12:12.8112394Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8112693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8112977Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8112996Z 
2022-12-01T11:12:12.8113260Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8113374Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8113394Z 
2022-12-01T11:12:12.8113501Z OK (skipped=1)
2022-12-01T11:12:12.8113520Z 
2022-12-01T11:12:12.8113644Z Generating XML reports...
2022-12-01T11:12:12.8114093Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110852.xml
2022-12-01T11:12:12.8114465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8114624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8115010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8115275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8115295Z 
2022-12-01T11:12:12.8115401Z Running tests...
2022-12-01T11:12:12.8115666Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8115978Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8116248Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8116267Z 
2022-12-01T11:12:12.8116532Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8116645Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8116665Z 
2022-12-01T11:12:12.8116754Z OK (skipped=1)
2022-12-01T11:12:12.8116792Z 
2022-12-01T11:12:12.8116899Z Generating XML reports...
2022-12-01T11:12:12.8117346Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110854.xml
2022-12-01T11:12:12.8117726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8117904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8118280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8118472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8118492Z 
2022-12-01T11:12:12.8118602Z Running tests...
2022-12-01T11:12:12.8118869Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8119166Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8119435Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8119454Z 
2022-12-01T11:12:12.8119722Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8119886Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8119906Z 
2022-12-01T11:12:12.8120019Z OK (skipped=1)
2022-12-01T11:12:12.8120038Z 
2022-12-01T11:12:12.8120162Z Generating XML reports...
2022-12-01T11:12:12.8120616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110855.xml
2022-12-01T11:12:12.8120990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8121168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8121536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8121730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8121750Z 
2022-12-01T11:12:12.8121859Z Running tests...
2022-12-01T11:12:12.8122131Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8122448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8122714Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8122733Z 
2022-12-01T11:12:12.8122996Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8123109Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8123129Z 
2022-12-01T11:12:12.8123218Z OK (skipped=1)
2022-12-01T11:12:12.8123256Z 
2022-12-01T11:12:12.8123361Z Generating XML reports...
2022-12-01T11:12:12.8123811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110857.xml
2022-12-01T11:12:12.8124184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8124361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8124806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8124997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8125016Z 
2022-12-01T11:12:12.8125125Z Running tests...
2022-12-01T11:12:12.8125390Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8125686Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8125957Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8125977Z 
2022-12-01T11:12:12.8126243Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8126357Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8126376Z 
2022-12-01T11:12:12.8126484Z OK (skipped=1)
2022-12-01T11:12:12.8126507Z 
2022-12-01T11:12:12.8126632Z Generating XML reports...
2022-12-01T11:12:12.8127080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110859.xml
2022-12-01T11:12:12.8127448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8127625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8127986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8128181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8128200Z 
2022-12-01T11:12:12.8128310Z Running tests...
2022-12-01T11:12:12.8128570Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8128885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8129200Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8129221Z 
2022-12-01T11:12:12.8129492Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8129605Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8129624Z 
2022-12-01T11:12:12.8129712Z OK (skipped=1)
2022-12-01T11:12:12.8129750Z 
2022-12-01T11:12:12.8129857Z Generating XML reports...
2022-12-01T11:12:12.8130309Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110901.xml
2022-12-01T11:12:12.8130684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8130861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8131244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8131443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8131463Z 
2022-12-01T11:12:12.8131575Z Running tests...
2022-12-01T11:12:12.8131841Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8132136Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8132394Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8132414Z 
2022-12-01T11:12:12.8132677Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8132793Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8132812Z 
2022-12-01T11:12:12.8132920Z OK (skipped=1)
2022-12-01T11:12:12.8132938Z 
2022-12-01T11:12:12.8133066Z Generating XML reports...
2022-12-01T11:12:12.8133513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110903.xml
2022-12-01T11:12:12.8133944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8134121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8134484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8134678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8134697Z 
2022-12-01T11:12:12.8134807Z Running tests...
2022-12-01T11:12:12.8135069Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8135381Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8135639Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8135659Z 
2022-12-01T11:12:12.8135920Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8136037Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8136056Z 
2022-12-01T11:12:12.8136149Z OK (skipped=1)
2022-12-01T11:12:12.8136188Z 
2022-12-01T11:12:12.8136293Z Generating XML reports...
2022-12-01T11:12:12.8136742Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110905.xml
2022-12-01T11:12:12.8137117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8137293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8137676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8137868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8137887Z 
2022-12-01T11:12:12.8137995Z Running tests...
2022-12-01T11:12:12.8138259Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8138606Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8138875Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8139099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113575
2022-12-01T11:12:12.8139319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113576
2022-12-01T11:12:12.8139694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8139868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8140249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8140442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8140806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8140969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8141342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8141530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8141778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8142027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8142431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8142830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8143111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8143323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8143671Z STAGE:2022-12-01 11:09:12 113575:113575 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8144431Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1640: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:12:12.8144552Z   warnings.warn(
2022-12-01T11:12:12.8144892Z STAGE:2022-12-01 11:09:17 113576:113576 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8145654Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1640: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:12:12.8145771Z   warnings.warn(
2022-12-01T11:12:12.8146118Z STAGE:2022-12-01 11:09:17 113576:113576 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8146462Z STAGE:2022-12-01 11:09:17 113575:113575 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8146792Z STAGE:2022-12-01 11:09:17 113575:113575 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8147107Z STAGE:2022-12-01 11:09:17 113575:113575 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8147440Z STAGE:2022-12-01 11:09:17 113576:113576 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8147779Z STAGE:2022-12-01 11:09:17 113576:113576 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8147882Z ok (10.119s)
2022-12-01T11:12:12.8147905Z 
2022-12-01T11:12:12.8148230Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8148351Z Ran 1 test in 10.119s
2022-12-01T11:12:12.8148370Z 
2022-12-01T11:12:12.8148464Z OK
2022-12-01T11:12:12.8148484Z 
2022-12-01T11:12:12.8148609Z Generating XML reports...
2022-12-01T11:12:12.8149067Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110907.xml
2022-12-01T11:12:12.8149649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8149837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8150229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8150422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8150442Z 
2022-12-01T11:12:12.8150557Z Running tests...
2022-12-01T11:12:12.8150829Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8151145Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8151411Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8151430Z 
2022-12-01T11:12:12.8151692Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8151786Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8151805Z 
2022-12-01T11:12:12.8151913Z OK (skipped=1)
2022-12-01T11:12:12.8151932Z 
2022-12-01T11:12:12.8152057Z Generating XML reports...
2022-12-01T11:12:12.8152511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110919.xml
2022-12-01T11:12:12.8152888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8153146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8153538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8153730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8153749Z 
2022-12-01T11:12:12.8153841Z Running tests...
2022-12-01T11:12:12.8154110Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8154429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8154706Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8154930Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113868
2022-12-01T11:12:12.8155152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113869
2022-12-01T11:12:12.8155532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8155712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8156097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8156271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8156634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8156808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8157182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8157372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8157620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8157931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8158348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8158727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8158959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8159187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8159526Z STAGE:2022-12-01 11:09:27 113868:113868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8159854Z STAGE:2022-12-01 11:09:27 113869:113869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8160199Z STAGE:2022-12-01 11:09:28 113868:113868 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8160542Z STAGE:2022-12-01 11:09:28 113869:113869 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8160875Z STAGE:2022-12-01 11:09:28 113868:113868 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8161200Z STAGE:2022-12-01 11:09:28 113869:113869 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8161515Z STAGE:2022-12-01 11:09:28 113868:113868 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8161855Z STAGE:2022-12-01 11:09:28 113869:113869 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8161959Z ok (7.656s)
2022-12-01T11:12:12.8161979Z 
2022-12-01T11:12:12.8162247Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8162361Z Ran 1 test in 7.656s
2022-12-01T11:12:12.8162381Z 
2022-12-01T11:12:12.8162539Z OK
2022-12-01T11:12:12.8162558Z 
2022-12-01T11:12:12.8162686Z Generating XML reports...
2022-12-01T11:12:12.8163148Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110921.xml
2022-12-01T11:12:12.8163523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8163682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8164064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8164258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8164277Z 
2022-12-01T11:12:12.8164387Z Running tests...
2022-12-01T11:12:12.8164653Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8164970Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8165245Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8165472Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114098
2022-12-01T11:12:12.8165676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114099
2022-12-01T11:12:12.8166052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8166229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8166610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8166802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8167169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8167351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8167776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8167974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8168205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8168454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8168855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8169254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8169485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8169721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8169827Z ok (6.674s)
2022-12-01T11:12:12.8169848Z 
2022-12-01T11:12:12.8170163Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8170261Z Ran 1 test in 6.674s
2022-12-01T11:12:12.8170300Z 
2022-12-01T11:12:12.8170374Z OK
2022-12-01T11:12:12.8170393Z 
2022-12-01T11:12:12.8170518Z Generating XML reports...
2022-12-01T11:12:12.8170974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110931.xml
2022-12-01T11:12:12.8171347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8171524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8171905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8172156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8172178Z 
2022-12-01T11:12:12.8172290Z Running tests...
2022-12-01T11:12:12.8172540Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8172859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8173120Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.003s)
2022-12-01T11:12:12.8173140Z 
2022-12-01T11:12:12.8173406Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8173521Z Ran 1 test in 0.003s
2022-12-01T11:12:12.8173542Z 
2022-12-01T11:12:12.8173653Z OK (skipped=1)
2022-12-01T11:12:12.8173672Z 
2022-12-01T11:12:12.8173799Z Generating XML reports...
2022-12-01T11:12:12.8174253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110940.xml
2022-12-01T11:12:12.8174631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8174792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8175177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8175375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8175394Z 
2022-12-01T11:12:12.8175505Z Running tests...
2022-12-01T11:12:12.8175772Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8176088Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8176348Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8176573Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114475
2022-12-01T11:12:12.8176783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114476
2022-12-01T11:12:12.8177238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8177423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8177810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8178004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8178369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8178542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8178920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8179118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8179350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8179598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8180004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8180404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8180634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8180866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8181210Z STAGE:2022-12-01 11:09:47 114475:114475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8181597Z STAGE:2022-12-01 11:09:47 114476:114476 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8181939Z STAGE:2022-12-01 11:09:47 114476:114476 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8182251Z STAGE:2022-12-01 11:09:47 114475:114475 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8182584Z STAGE:2022-12-01 11:09:48 114476:114476 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8182908Z STAGE:2022-12-01 11:09:48 114475:114475 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8183460Z STAGE:2022-12-01 11:09:48 114475:114475 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:09:48 114476:114476 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8183480Z 
2022-12-01T11:12:12.8183584Z ok (5.888s)
2022-12-01T11:12:12.8183603Z 
2022-12-01T11:12:12.8183871Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8183991Z Ran 1 test in 5.888s
2022-12-01T11:12:12.8184013Z 
2022-12-01T11:12:12.8184108Z OK
2022-12-01T11:12:12.8184127Z 
2022-12-01T11:12:12.8184253Z Generating XML reports...
2022-12-01T11:12:12.8184689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110942.xml
2022-12-01T11:12:12.8185063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8185242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8185624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8185818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8185839Z 
2022-12-01T11:12:12.8185948Z Running tests...
2022-12-01T11:12:12.8186215Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8186583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8186839Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8187059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114735
2022-12-01T11:12:12.8187282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114736
2022-12-01T11:12:12.8187660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8187838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8188215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8188408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8188784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8188960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8189312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8189733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8189989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8190237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8190648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8191050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8191365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8191594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8191939Z STAGE:2022-12-01 11:09:55 114736:114736 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8192253Z STAGE:2022-12-01 11:09:55 114735:114735 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8192593Z STAGE:2022-12-01 11:09:55 114736:114736 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8192928Z STAGE:2022-12-01 11:09:55 114735:114735 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8193261Z STAGE:2022-12-01 11:09:56 114736:114736 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8193599Z STAGE:2022-12-01 11:09:56 114736:114736 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8193938Z STAGE:2022-12-01 11:09:56 114735:114735 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8194276Z STAGE:2022-12-01 11:09:56 114735:114735 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8194378Z ok (5.981s)
2022-12-01T11:12:12.8194398Z 
2022-12-01T11:12:12.8194667Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8194764Z Ran 1 test in 5.981s
2022-12-01T11:12:12.8194784Z 
2022-12-01T11:12:12.8194878Z OK
2022-12-01T11:12:12.8194898Z 
2022-12-01T11:12:12.8195025Z Generating XML reports...
2022-12-01T11:12:12.8195484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110950.xml
2022-12-01T11:12:12.8195864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8196041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8196485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8196686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8196705Z 
2022-12-01T11:12:12.8196796Z Running tests...
2022-12-01T11:12:12.8197065Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8197382Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8197647Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8197667Z 
2022-12-01T11:12:12.8197924Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8198037Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8198057Z 
2022-12-01T11:12:12.8198165Z OK (skipped=1)
2022-12-01T11:12:12.8198184Z 
2022-12-01T11:12:12.8198309Z Generating XML reports...
2022-12-01T11:12:12.8198772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110958.xml
2022-12-01T11:12:12.8199129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8199307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8199691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8199883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8199902Z 
2022-12-01T11:12:12.8200012Z Running tests...
2022-12-01T11:12:12.8200279Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8200594Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8200846Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8200912Z 
2022-12-01T11:12:12.8201182Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8201277Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8201297Z 
2022-12-01T11:12:12.8201405Z OK (skipped=1)
2022-12-01T11:12:12.8201425Z 
2022-12-01T11:12:12.8201551Z Generating XML reports...
2022-12-01T11:12:12.8202004Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111000.xml
2022-12-01T11:12:12.8202379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8202555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8202937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8203129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8203152Z 
2022-12-01T11:12:12.8203242Z Running tests...
2022-12-01T11:12:12.8203513Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8203832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8204098Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8204117Z 
2022-12-01T11:12:12.8204376Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8204489Z Ran 1 test in 0.003s
2022-12-01T11:12:12.8204508Z 
2022-12-01T11:12:12.8204617Z OK (skipped=1)
2022-12-01T11:12:12.8204636Z 
2022-12-01T11:12:12.8204762Z Generating XML reports...
2022-12-01T11:12:12.8205216Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111002.xml
2022-12-01T11:12:12.8205574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8205800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8206197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8206390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8206409Z 
2022-12-01T11:12:12.8206518Z Running tests...
2022-12-01T11:12:12.8206782Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8207095Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8207362Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8207381Z 
2022-12-01T11:12:12.8207643Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8207738Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8207761Z 
2022-12-01T11:12:12.8207870Z OK (skipped=1)
2022-12-01T11:12:12.8207892Z 
2022-12-01T11:12:12.8208018Z Generating XML reports...
2022-12-01T11:12:12.8208469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111004.xml
2022-12-01T11:12:12.8208842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8209021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8209403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8209597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8209616Z 
2022-12-01T11:12:12.8209706Z Running tests...
2022-12-01T11:12:12.8209971Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8210343Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8210599Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8210820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115220
2022-12-01T11:12:12.8211045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115221
2022-12-01T11:12:12.8211418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8211597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8211981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8212157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8212523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8212706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8213081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8213273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8213524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8213775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8214183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8214585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8214797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8215088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8215439Z STAGE:2022-12-01 11:10:11 115220:115220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8215766Z STAGE:2022-12-01 11:10:12 115221:115221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8216102Z STAGE:2022-12-01 11:10:13 115220:115220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8216433Z STAGE:2022-12-01 11:10:13 115221:115221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8216770Z STAGE:2022-12-01 11:10:13 115221:115221 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8217097Z STAGE:2022-12-01 11:10:13 115220:115220 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8217417Z STAGE:2022-12-01 11:10:13 115221:115221 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8217764Z STAGE:2022-12-01 11:10:13 115220:115220 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8217867Z ok (7.634s)
2022-12-01T11:12:12.8217887Z 
2022-12-01T11:12:12.8218158Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8218272Z Ran 1 test in 7.634s
2022-12-01T11:12:12.8218292Z 
2022-12-01T11:12:12.8218385Z OK
2022-12-01T11:12:12.8218404Z 
2022-12-01T11:12:12.8218532Z Generating XML reports...
2022-12-01T11:12:12.8218989Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111006.xml
2022-12-01T11:12:12.8219363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8219522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8219906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8220157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8220177Z 
2022-12-01T11:12:12.8220286Z Running tests...
2022-12-01T11:12:12.8220552Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8220869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8221136Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8221359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115558
2022-12-01T11:12:12.8221561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115559
2022-12-01T11:12:12.8221937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8222112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8222497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8222691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8223056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8223232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8223609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8223803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8224030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8224277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8224733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8225139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8225369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8225599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8225939Z STAGE:2022-12-01 11:10:21 115559:115559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8226271Z STAGE:2022-12-01 11:10:21 115558:115558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8226612Z STAGE:2022-12-01 11:10:23 115558:115558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8226927Z STAGE:2022-12-01 11:10:23 115559:115559 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8227278Z STAGE:2022-12-01 11:10:23 115558:115558 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8227604Z STAGE:2022-12-01 11:10:23 115559:115559 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8227936Z STAGE:2022-12-01 11:10:23 115559:115559 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8228273Z STAGE:2022-12-01 11:10:23 115558:115558 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8228375Z ok (7.682s)
2022-12-01T11:12:12.8228395Z 
2022-12-01T11:12:12.8228663Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8228779Z Ran 1 test in 7.682s
2022-12-01T11:12:12.8228799Z 
2022-12-01T11:12:12.8228877Z OK
2022-12-01T11:12:12.8228915Z 
2022-12-01T11:12:12.8229022Z Generating XML reports...
2022-12-01T11:12:12.8229701Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111016.xml
2022-12-01T11:12:12.8230187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8230365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8230749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8230945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8230964Z 
2022-12-01T11:12:12.8231076Z Running tests...
2022-12-01T11:12:12.8231340Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8231643Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8231913Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8231937Z 
2022-12-01T11:12:12.8232204Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8232319Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8232338Z 
2022-12-01T11:12:12.8232447Z OK (skipped=1)
2022-12-01T11:12:12.8232466Z 
2022-12-01T11:12:12.8232592Z Generating XML reports...
2022-12-01T11:12:12.8233043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111026.xml
2022-12-01T11:12:12.8233414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8233590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8233955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8234149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8234168Z 
2022-12-01T11:12:12.8234281Z Running tests...
2022-12-01T11:12:12.8234601Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8234928Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8235192Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2022-12-01T11:12:12.8235212Z 
2022-12-01T11:12:12.8235474Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8235588Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8235607Z 
2022-12-01T11:12:12.8235697Z OK (skipped=1)
2022-12-01T11:12:12.8235737Z 
2022-12-01T11:12:12.8235845Z Generating XML reports...
2022-12-01T11:12:12.8236297Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111028.xml
2022-12-01T11:12:12.8236672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8236855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8237239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8237432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8237451Z 
2022-12-01T11:12:12.8237560Z Running tests...
2022-12-01T11:12:12.8237825Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8238122Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8238509Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2022-12-01T11:12:12.8238529Z 
2022-12-01T11:12:12.8238791Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8238905Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8238924Z 
2022-12-01T11:12:12.8239082Z OK (skipped=1)
2022-12-01T11:12:12.8239102Z 
2022-12-01T11:12:12.8239229Z Generating XML reports...
2022-12-01T11:12:12.8239682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111030.xml
2022-12-01T11:12:12.8240060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8240239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8240605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8240803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8240823Z 
2022-12-01T11:12:12.8240932Z Running tests...
2022-12-01T11:12:12.8241196Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8241513Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8241786Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T11:12:12.8241806Z 
2022-12-01T11:12:12.8242070Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8242181Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8242201Z 
2022-12-01T11:12:12.8242309Z OK (skipped=1)
2022-12-01T11:12:12.8242327Z 
2022-12-01T11:12:12.8242434Z Generating XML reports...
2022-12-01T11:12:12.8242885Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111032.xml
2022-12-01T11:12:12.8243260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8243440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8243822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8244063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8244085Z 
2022-12-01T11:12:12.8244198Z Running tests...
2022-12-01T11:12:12.8244463Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8244758Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8245045Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-12-01T11:12:12.8245065Z 
2022-12-01T11:12:12.8245332Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8245444Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8245463Z 
2022-12-01T11:12:12.8245572Z OK (skipped=1)
2022-12-01T11:12:12.8245591Z 
2022-12-01T11:12:12.8245719Z Generating XML reports...
2022-12-01T11:12:12.8246169Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111034.xml
2022-12-01T11:12:12.8246548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8246724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8247087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8247282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8247301Z 
2022-12-01T11:12:12.8247413Z Running tests...
2022-12-01T11:12:12.8247679Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8247995Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8248310Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-12-01T11:12:12.8248376Z 
2022-12-01T11:12:12.8248651Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8248765Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8248784Z 
2022-12-01T11:12:12.8248894Z OK (skipped=1)
2022-12-01T11:12:12.8248912Z 
2022-12-01T11:12:12.8249017Z Generating XML reports...
2022-12-01T11:12:12.8249471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111036.xml
2022-12-01T11:12:12.8249845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8250022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8250403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8250595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8250618Z 
2022-12-01T11:12:12.8250727Z Running tests...
2022-12-01T11:12:12.8250992Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8251310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8251597Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2022-12-01T11:12:12.8251617Z 
2022-12-01T11:12:12.8251880Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8251996Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8252015Z 
2022-12-01T11:12:12.8252124Z OK (skipped=1)
2022-12-01T11:12:12.8252142Z 
2022-12-01T11:12:12.8252269Z Generating XML reports...
2022-12-01T11:12:12.8252719Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111038.xml
2022-12-01T11:12:12.8253090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8253315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8253709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8253885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8253904Z 
2022-12-01T11:12:12.8254012Z Running tests...
2022-12-01T11:12:12.8254280Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8254597Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8254890Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T11:12:12.8254909Z 
2022-12-01T11:12:12.8255172Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8255289Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8255308Z 
2022-12-01T11:12:12.8255416Z OK (skipped=1)
2022-12-01T11:12:12.8255438Z 
2022-12-01T11:12:12.8255544Z Generating XML reports...
2022-12-01T11:12:12.8255994Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111040.xml
2022-12-01T11:12:12.8256370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8256549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8256931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8257124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8257144Z 
2022-12-01T11:12:12.8257253Z Running tests...
2022-12-01T11:12:12.8257516Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8257888Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8258128Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8258350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116274
2022-12-01T11:12:12.8258572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116275
2022-12-01T11:12:12.8258945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8259121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8259506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8259701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8260066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8260250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8260608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8260800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8261048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8261297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8261703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8262104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8262337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8262616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8262708Z ok (5.658s)
2022-12-01T11:12:12.8262744Z 
2022-12-01T11:12:12.8262996Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8263109Z Ran 1 test in 5.658s
2022-12-01T11:12:12.8263130Z 
2022-12-01T11:12:12.8263225Z OK
2022-12-01T11:12:12.8263244Z 
2022-12-01T11:12:12.8263372Z Generating XML reports...
2022-12-01T11:12:12.8263831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111042.xml
2022-12-01T11:12:12.8264204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8264379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8264759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8264940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8264960Z 
2022-12-01T11:12:12.8265070Z Running tests...
2022-12-01T11:12:12.8265336Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8265653Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8265939Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8266163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116504
2022-12-01T11:12:12.8266387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116505
2022-12-01T11:12:12.8266761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8266922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8267367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8267558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8267926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8268104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8268485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8268681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8268931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8269177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8269807Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8270258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8270489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8270721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8271064Z STAGE:2022-12-01 11:10:55 116504:116504 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8271393Z STAGE:2022-12-01 11:10:55 116505:116505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8271737Z STAGE:2022-12-01 11:10:55 116505:116505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8272077Z STAGE:2022-12-01 11:10:55 116504:116504 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8272185Z ok (5.947s)
2022-12-01T11:12:12.8272277Z 
2022-12-01T11:12:12.8272540Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8272654Z Ran 1 test in 5.947s
2022-12-01T11:12:12.8272673Z 
2022-12-01T11:12:12.8272766Z OK
2022-12-01T11:12:12.8272785Z 
2022-12-01T11:12:12.8272911Z Generating XML reports...
2022-12-01T11:12:12.8273366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111050.xml
2022-12-01T11:12:12.8273743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8273984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8274611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8274792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8274836Z 
2022-12-01T11:12:12.8274931Z Running tests...
2022-12-01T11:12:12.8275201Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8275516Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8275792Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8276013Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116694
2022-12-01T11:12:12.8276235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116695
2022-12-01T11:12:12.8276609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8276788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8277154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8277439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8277812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8277989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8278366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8278555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8278803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8279052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8279435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8279842Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8280074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8280307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8280649Z STAGE:2022-12-01 11:11:03 116694:116694 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8280979Z STAGE:2022-12-01 11:11:03 116695:116695 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:12:12.8281321Z STAGE:2022-12-01 11:11:03 116695:116695 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8281656Z STAGE:2022-12-01 11:11:03 116694:116694 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:12:12.8281763Z ok (6.057s)
2022-12-01T11:12:12.8281786Z 
2022-12-01T11:12:12.8282104Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8282228Z Ran 1 test in 6.057s
2022-12-01T11:12:12.8282248Z 
2022-12-01T11:12:12.8282339Z OK
2022-12-01T11:12:12.8282358Z 
2022-12-01T11:12:12.8282482Z Generating XML reports...
2022-12-01T11:12:12.8282940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111058.xml
2022-12-01T11:12:12.8283315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8283493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8283875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8284070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8284090Z 
2022-12-01T11:12:12.8284180Z Running tests...
2022-12-01T11:12:12.8284451Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8284771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8285059Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T11:12:12.8285079Z 
2022-12-01T11:12:12.8285341Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8285460Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8285479Z 
2022-12-01T11:12:12.8285590Z OK (skipped=1)
2022-12-01T11:12:12.8285609Z 
2022-12-01T11:12:12.8285735Z Generating XML reports...
2022-12-01T11:12:12.8286169Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111106.xml
2022-12-01T11:12:12.8286544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8286773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8287159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8287349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8287368Z 
2022-12-01T11:12:12.8287477Z Running tests...
2022-12-01T11:12:12.8287746Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8288061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8288338Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T11:12:12.8288357Z 
2022-12-01T11:12:12.8288605Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8288719Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8288738Z 
2022-12-01T11:12:12.8288850Z OK (skipped=1)
2022-12-01T11:12:12.8288869Z 
2022-12-01T11:12:12.8288998Z Generating XML reports...
2022-12-01T11:12:12.8289450Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111108.xml
2022-12-01T11:12:12.8289823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8290000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8290382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8290575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8290595Z 
2022-12-01T11:12:12.8290688Z Running tests...
2022-12-01T11:12:12.8290956Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8291272Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8291625Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T11:12:12.8291646Z 
2022-12-01T11:12:12.8291912Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8292026Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8292045Z 
2022-12-01T11:12:12.8292153Z OK (skipped=1)
2022-12-01T11:12:12.8292171Z 
2022-12-01T11:12:12.8292296Z Generating XML reports...
2022-12-01T11:12:12.8292746Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111110.xml
2022-12-01T11:12:12.8293100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8293278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8293662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8293861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8293880Z 
2022-12-01T11:12:12.8293989Z Running tests...
2022-12-01T11:12:12.8294262Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8294580Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8294876Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2022-12-01T11:12:12.8294897Z 
2022-12-01T11:12:12.8295163Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8295260Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8295279Z 
2022-12-01T11:12:12.8295388Z OK (skipped=1)
2022-12-01T11:12:12.8295406Z 
2022-12-01T11:12:12.8295530Z Generating XML reports...
2022-12-01T11:12:12.8295981Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111112.xml
2022-12-01T11:12:12.8296413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8296587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8296968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8297161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8297180Z 
2022-12-01T11:12:12.8297289Z Running tests...
2022-12-01T11:12:12.8297538Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8297850Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8298136Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T11:12:12.8298159Z 
2022-12-01T11:12:12.8298422Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8298534Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8298553Z 
2022-12-01T11:12:12.8298660Z OK (skipped=1)
2022-12-01T11:12:12.8298679Z 
2022-12-01T11:12:12.8298804Z Generating XML reports...
2022-12-01T11:12:12.8299253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111114.xml
2022-12-01T11:12:12.8299608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8299787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8300169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8300362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8300386Z 
2022-12-01T11:12:12.8300495Z Running tests...
2022-12-01T11:12:12.8300806Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8301130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8301423Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2022-12-01T11:12:12.8301443Z 
2022-12-01T11:12:12.8301706Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8301800Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8301819Z 
2022-12-01T11:12:12.8301931Z OK (skipped=1)
2022-12-01T11:12:12.8301951Z 
2022-12-01T11:12:12.8302074Z Generating XML reports...
2022-12-01T11:12:12.8302526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111116.xml
2022-12-01T11:12:12.8302896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8303079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8303461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8303656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8303675Z 
2022-12-01T11:12:12.8303785Z Running tests...
2022-12-01T11:12:12.8304031Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8304345Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8304614Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8304839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117384
2022-12-01T11:12:12.8305061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117385
2022-12-01T11:12:12.8305492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8305668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8306051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8306225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8306593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8306772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8307150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8307339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8307590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8307839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8308243Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8308644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8308857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8309086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8309193Z ok (6.948s)
2022-12-01T11:12:12.8309212Z 
2022-12-01T11:12:12.8309843Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8310041Z Ran 1 test in 6.948s
2022-12-01T11:12:12.8310066Z 
2022-12-01T11:12:12.8310160Z OK
2022-12-01T11:12:12.8310180Z 
2022-12-01T11:12:12.8310393Z Generating XML reports...
2022-12-01T11:12:12.8310879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111118.xml
2022-12-01T11:12:12.8311235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8311412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8311795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8311993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8312013Z 
2022-12-01T11:12:12.8312123Z Running tests...
2022-12-01T11:12:12.8312391Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8312707Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8312994Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2022-12-01T11:12:12.8313013Z 
2022-12-01T11:12:12.8313275Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8313369Z Ran 1 test in 0.002s
2022-12-01T11:12:12.8313388Z 
2022-12-01T11:12:12.8313498Z OK (skipped=1)
2022-12-01T11:12:12.8313517Z 
2022-12-01T11:12:12.8313644Z Generating XML reports...
2022-12-01T11:12:12.8314098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111127.xml
2022-12-01T11:12:12.8314474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8314651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8315035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8315309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8315329Z 
2022-12-01T11:12:12.8315440Z Running tests...
2022-12-01T11:12:12.8315688Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8315999Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8316250Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8316473Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117773
2022-12-01T11:12:12.8316692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117774
2022-12-01T11:12:12.8317063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8317241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8317628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8317802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8318171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8318347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8318722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8318912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8319162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8319408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8319861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8320272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8320484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8320715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8320976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpov7fwkhh
2022-12-01T11:12:12.8321252Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpov7fwkhh/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8321510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2svbwpnq
2022-12-01T11:12:12.8321779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2svbwpnq/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8321888Z ok (6.689s)
2022-12-01T11:12:12.8321908Z 
2022-12-01T11:12:12.8322182Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8322279Z Ran 1 test in 6.689s
2022-12-01T11:12:12.8322317Z 
2022-12-01T11:12:12.8322391Z OK
2022-12-01T11:12:12.8322410Z 
2022-12-01T11:12:12.8322531Z Generating XML reports...
2022-12-01T11:12:12.8323043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111129.xml
2022-12-01T11:12:12.8323419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8323594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8323972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8324166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8324269Z 
2022-12-01T11:12:12.8324387Z Running tests...
2022-12-01T11:12:12.8324640Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8324956Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8325254Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8325475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118012
2022-12-01T11:12:12.8325693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118013
2022-12-01T11:12:12.8326066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8326242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8326623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8326823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8327169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8327344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8327718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8327907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8328155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8328399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8328802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8329255Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8329476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8329707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8329966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsicdwtbf
2022-12-01T11:12:12.8330239Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsicdwtbf/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8330495Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz1qi_ig
2022-12-01T11:12:12.8330763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz1qi_ig/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8331547Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.8332327Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:12:12.8332478Z ok (7.181s)
2022-12-01T11:12:12.8332498Z 
2022-12-01T11:12:12.8332772Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8332886Z Ran 1 test in 7.181s
2022-12-01T11:12:12.8332906Z 
2022-12-01T11:12:12.8332999Z OK
2022-12-01T11:12:12.8333018Z 
2022-12-01T11:12:12.8333142Z Generating XML reports...
2022-12-01T11:12:12.8333578Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111138.xml
2022-12-01T11:12:12.8333953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8334132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8334516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8334710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8334732Z 
2022-12-01T11:12:12.8334843Z Running tests...
2022-12-01T11:12:12.8335112Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8335432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8335699Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8335921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118313
2022-12-01T11:12:12.8336142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118314
2022-12-01T11:12:12.8336519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8336697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8337077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8337273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8337687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8337869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8338235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8338423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8338671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8338917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8339325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8339736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8339969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8340198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8340439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.8340665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.8341069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.8341466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.8341708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.8342001Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.8342393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.8342789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.8343048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg8f1xgg
2022-12-01T11:12:12.8343321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg8f1xgg/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8343557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3gu8aym3
2022-12-01T11:12:12.8343826Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3gu8aym3/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8343930Z ok (10.958s)
2022-12-01T11:12:12.8343953Z 
2022-12-01T11:12:12.8344231Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8344346Z Ran 1 test in 10.959s
2022-12-01T11:12:12.8344365Z 
2022-12-01T11:12:12.8344459Z OK
2022-12-01T11:12:12.8344479Z 
2022-12-01T11:12:12.8344603Z Generating XML reports...
2022-12-01T11:12:12.8345056Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111148.xml
2022-12-01T11:12:12.8345410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8345588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8345972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8346164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8346183Z 
2022-12-01T11:12:12.8346297Z Running tests...
2022-12-01T11:12:12.8346634Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8346967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2022-12-01T11:12:12.8347254Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:12:12.8347478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118653
2022-12-01T11:12:12.8347682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118654
2022-12-01T11:12:12.8348054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8348233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8348617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8348813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8349182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:12:12.8349586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:12:12.8349994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:12:12.8350165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:12:12.8350417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:12:12.8350663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:12:12.8351066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8351463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:12:12.8351785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:12:12.8352014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:12:12.8352257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:12:12.8352499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:12:12.8352882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.8353278Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:12:12.8353521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:12:12.8353769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:12:12.8354162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.8354558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:12:12.8354819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeq9__iw7
2022-12-01T11:12:12.8355094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeq9__iw7/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8355352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ztq1p95
2022-12-01T11:12:12.8355605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ztq1p95/_remote_module_non_scriptable.py
2022-12-01T11:12:12.8355712Z ok (10.604s)
2022-12-01T11:12:12.8355736Z 
2022-12-01T11:12:12.8356011Z ----------------------------------------------------------------------
2022-12-01T11:12:12.8356187Z Ran 1 test in 10.604s
2022-12-01T11:12:12.8356208Z 
2022-12-01T11:12:12.8356307Z OK
2022-12-01T11:12:12.8356327Z 
2022-12-01T11:12:12.8356456Z Generating XML reports...
2022-12-01T11:12:12.8356913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111201.xml
2022-12-01T11:12:12.8356932Z 
2022-12-01T11:12:12.8357381Z ##[endgroup]
2022-12-01T11:12:12.8357835Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_0c81ynry)
2022-12-01T11:12:12.8357854Z 
2022-12-01T11:15:25.6412554Z 
2022-12-01T11:15:25.6415083Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2022-12-01T11:15:25.6416026Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_7nnidt1g)
2022-12-01T11:15:25.6422805Z <unittest.suite.TestSuite tests=[]>
2022-12-01T11:15:25.6469814Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2022-12-01T11:15:25.6530708Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6534327Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6535065Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6535867Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6536682Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6537544Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6538451Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6539381Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6540301Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6541260Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6542299Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6543262Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6544254Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6545180Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6546097Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6546997Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6547820Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6548637Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6550082Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6550882Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6551760Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6552590Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6553357Z test_all_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6554061Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6554830Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6555628Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6556416Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6557176Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6557928Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6558615Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6559316Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6560038Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6560742Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6561467Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6562278Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6563016Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6563762Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6564543Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6565320Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6566049Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6566915Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6567737Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6568595Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6569419Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6570249Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6571024Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6571842Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6572596Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6573325Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6574167Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6574949Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6575739Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6576477Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6577279Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6578038Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6578795Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6579530Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6580256Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6580967Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6581721Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6582600Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6583319Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6584035Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6584738Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6585389Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6586129Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6586887Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6587588Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6588286Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6588959Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6590221Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6590935Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6591675Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6592393Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6593087Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6593756Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6594450Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6595174Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6595909Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6596624Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6597347Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6598052Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6598809Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6599697Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6600551Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6601337Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6602183Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6603006Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6603782Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6604590Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6605407Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6606222Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6607011Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6607846Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6608655Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6609534Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6610341Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6611154Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6611962Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6612662Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6613368Z test_backend_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6614015Z test_barrier (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6614737Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6615398Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6616099Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6616802Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6617498Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6618213Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6618940Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6619597Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6620315Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6621072Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6621868Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6622753Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6623397Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6623814Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6624210Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6624638Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6625063Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6625460Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6625846Z test_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6626215Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6626605Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6626978Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6627433Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6627840Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6628277Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6628792Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6629245Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6630283Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6630686Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6631122Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6631572Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6632023Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6632475Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6632902Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6633346Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6633739Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6634110Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6634501Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6634894Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6635303Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6635742Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6636163Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6636689Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6637103Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6637567Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6638043Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6638611Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6639224Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6639828Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6640431Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6641033Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6641638Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6642239Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6642823Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6643368Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6643863Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6644377Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6644765Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6645164Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6645574Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6645957Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6646378Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6646818Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6647264Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6647732Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6648145Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6648544Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6648943Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6649382Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6649813Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6650210Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6650628Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6651053Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6651482Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6651883Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6652284Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6652756Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6653158Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6653565Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6653984Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6654424Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6654848Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6655225Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6655622Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6656027Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6656442Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6656825Z test_gather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6657176Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6657546Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6657925Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6658288Z test_gather_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6658658Z test_gather_object (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6659043Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6659426Z test_get_backend (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6659771Z test_get_future (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6660127Z test_get_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6660504Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6660881Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6661276Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6661651Z test_irecv (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6662033Z test_isend (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6662421Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6662824Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6663235Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6663670Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6664119Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6664535Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6664939Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6665371Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6665805Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6666215Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6666632Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6667048Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6667457Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6667900Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6668294Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6668692Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6669140Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6670237Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6670810Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6671267Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6671711Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6672163Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6672594Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6672994Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6673424Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6673862Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6674311Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6674772Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6675294Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6675780Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6676183Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6676579Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6676984Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6677386Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6677755Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6678135Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6678525Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6678897Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6679271Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6679699Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6680070Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6680450Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6680844Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6681249Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6681608Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6681977Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6682360Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6682728Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6683090Z test_scatter (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6683453Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6683817Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6684194Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6684576Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6684951Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6685331Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6685712Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6686078Z test_send_recv (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6686434Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6686851Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6717711Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6718184Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6718744Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6719174Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6719613Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6720038Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6720425Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6720846Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6721291Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6721697Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6722109Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6722589Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6722995Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6723374Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6723807Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6724266Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6724703Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.6725442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6725909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6726508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6726975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6727210Z 
2022-12-01T11:15:25.6727320Z Running tests...
2022-12-01T11:15:25.6727732Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6728335Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6728955Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6729536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62025
2022-12-01T11:15:25.6730000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62026
2022-12-01T11:15:25.6730611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6731070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6731655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6732122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6732724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6733180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6733759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6734214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6734676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6735187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6735861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6736554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6737177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6737662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6738184Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6739046Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6739731Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6740561Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6741246Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6742069Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6742745Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6743577Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6744253Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6744818Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6745658Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6746636Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6747322Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6748132Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6748802Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2022-12-01T11:15:25.6750221Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2022-12-01T11:15:25.6750739Z ok (7.069s)
2022-12-01T11:15:25.6750872Z 
2022-12-01T11:15:25.6751157Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6751487Z Ran 1 test in 7.069s
2022-12-01T11:15:25.6751650Z 
2022-12-01T11:15:25.6751744Z OK
2022-12-01T11:15:25.6751880Z 
2022-12-01T11:15:25.6751988Z Generating XML reports...
2022-12-01T11:15:25.6752613Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104823.xml
2022-12-01T11:15:25.6753346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6753805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6754374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6754853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6755194Z 
2022-12-01T11:15:25.6755306Z Running tests...
2022-12-01T11:15:25.6755706Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6756248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6756791Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.004s)
2022-12-01T11:15:25.6757106Z 
2022-12-01T11:15:25.6757369Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6757680Z Ran 1 test in 0.004s
2022-12-01T11:15:25.6757841Z 
2022-12-01T11:15:25.6757949Z OK (skipped=1)
2022-12-01T11:15:25.6758104Z 
2022-12-01T11:15:25.6758228Z Generating XML reports...
2022-12-01T11:15:25.6758827Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104832.xml
2022-12-01T11:15:25.6759556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6760020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6760601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6761067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6761300Z 
2022-12-01T11:15:25.6761407Z Running tests...
2022-12-01T11:15:25.6761813Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6762345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6762844Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6763343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62463
2022-12-01T11:15:25.6763803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62464
2022-12-01T11:15:25.6764482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6764955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6765544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6766021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6766594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6767042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6767617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6768129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6768593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6769110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6769776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6770465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6771000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6771478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6771828Z ok (3.930s)
2022-12-01T11:15:25.6771961Z 
2022-12-01T11:15:25.6772233Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6772562Z Ran 1 test in 3.931s
2022-12-01T11:15:25.6772723Z 
2022-12-01T11:15:25.6772887Z OK
2022-12-01T11:15:25.6773024Z 
2022-12-01T11:15:25.6773130Z Generating XML reports...
2022-12-01T11:15:25.6773756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104834.xml
2022-12-01T11:15:25.6774484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6774940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6775507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6775983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6776216Z 
2022-12-01T11:15:25.6776324Z Running tests...
2022-12-01T11:15:25.6776709Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6777248Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6777793Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6778864Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.701s)
2022-12-01T11:15:25.6779407Z 
2022-12-01T11:15:25.6779672Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6779984Z Ran 1 test in 1.701s
2022-12-01T11:15:25.6780144Z 
2022-12-01T11:15:25.6780251Z OK (skipped=1)
2022-12-01T11:15:25.6780406Z 
2022-12-01T11:15:25.6780531Z Generating XML reports...
2022-12-01T11:15:25.6781130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104841.xml
2022-12-01T11:15:25.6781959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6782426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6783009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6783471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6783705Z 
2022-12-01T11:15:25.6783811Z Running tests...
2022-12-01T11:15:25.6784216Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6784736Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6785289Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6785818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62911
2022-12-01T11:15:25.6786283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62912
2022-12-01T11:15:25.6786889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6787352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6787937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6788416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6788985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6789929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6790573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6791028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6791602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6792113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6792787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6793474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6794010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6794520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpal700alx
2022-12-01T11:15:25.6795078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpal700alx/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6795579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6796096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp07lr6pgl
2022-12-01T11:15:25.6796643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp07lr6pgl/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6797149Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6797644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6798005Z ok (4.122s)
2022-12-01T11:15:25.6798156Z 
2022-12-01T11:15:25.6798432Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6798746Z Ran 1 test in 4.122s
2022-12-01T11:15:25.6798907Z 
2022-12-01T11:15:25.6799001Z OK
2022-12-01T11:15:25.6799134Z 
2022-12-01T11:15:25.6799257Z Generating XML reports...
2022-12-01T11:15:25.6799856Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104845.xml
2022-12-01T11:15:25.6800663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6801137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6801724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6802191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6802424Z 
2022-12-01T11:15:25.6802533Z Running tests...
2022-12-01T11:15:25.6802940Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6803459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6804029Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6804577Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63244
2022-12-01T11:15:25.6805040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63245
2022-12-01T11:15:25.6805643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6806101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6806683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6807145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6807735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6808186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6808761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6809284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6809750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6810261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6810935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6811623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6812157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6812640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6813132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3q3n8y5
2022-12-01T11:15:25.6813675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiy2fqfzi
2022-12-01T11:15:25.6814219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3q3n8y5/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6814772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiy2fqfzi/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6815280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6815775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6816134Z ok (4.013s)
2022-12-01T11:15:25.6816284Z 
2022-12-01T11:15:25.6816554Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6816867Z Ran 1 test in 4.013s
2022-12-01T11:15:25.6817028Z 
2022-12-01T11:15:25.6817120Z OK
2022-12-01T11:15:25.6817256Z 
2022-12-01T11:15:25.6817379Z Generating XML reports...
2022-12-01T11:15:25.6818033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104851.xml
2022-12-01T11:15:25.6818773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6819234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6819804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6820287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6820520Z 
2022-12-01T11:15:25.6820628Z Running tests...
2022-12-01T11:15:25.6821035Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6821556Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6822117Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6822663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63484
2022-12-01T11:15:25.6823123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63485
2022-12-01T11:15:25.6823728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6824183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6824763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6825228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6825816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6826269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6826910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6827374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6827833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6828344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6828993Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6830332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6830874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6831356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6831860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuv8a4hmy
2022-12-01T11:15:25.6832412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuv8a4hmy/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6832958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpey9qg8oo
2022-12-01T11:15:25.6833502Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpey9qg8oo/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6834006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6834502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6834995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6835469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6835962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6836537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6837038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6837505Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6837986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6838463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6838928Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6839415Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6839764Z ok (6.974s)
2022-12-01T11:15:25.6839914Z 
2022-12-01T11:15:25.6840197Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6840518Z Ran 1 test in 6.974s
2022-12-01T11:15:25.6840683Z 
2022-12-01T11:15:25.6840774Z OK
2022-12-01T11:15:25.6840910Z 
2022-12-01T11:15:25.6841035Z Generating XML reports...
2022-12-01T11:15:25.6841640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104857.xml
2022-12-01T11:15:25.6842373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6842829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6843412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6843874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6844106Z 
2022-12-01T11:15:25.6844214Z Running tests...
2022-12-01T11:15:25.6844624Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6845230Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6845803Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6846351Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63836
2022-12-01T11:15:25.6846813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63837
2022-12-01T11:15:25.6847415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6847874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6848459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6848940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6849522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6849978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6850556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6851016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6851479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6851987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6852649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6853333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6853916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6854405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6854918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjd7dgrfu
2022-12-01T11:15:25.6855454Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjd7dgrfu/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6855999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzcrxa75
2022-12-01T11:15:25.6856547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzcrxa75/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6857056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6857548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6858044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6858537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6858874Z ok (5.850s)
2022-12-01T11:15:25.6859024Z 
2022-12-01T11:15:25.6859298Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6859624Z Ran 1 test in 5.851s
2022-12-01T11:15:25.6859787Z 
2022-12-01T11:15:25.6859861Z OK
2022-12-01T11:15:25.6859996Z 
2022-12-01T11:15:25.6860119Z Generating XML reports...
2022-12-01T11:15:25.6860737Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104906.xml
2022-12-01T11:15:25.6861466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6861914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6862504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6863097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6863331Z 
2022-12-01T11:15:25.6863422Z Running tests...
2022-12-01T11:15:25.6863828Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6864363Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6864948Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6865491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64177
2022-12-01T11:15:25.6865957Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64178
2022-12-01T11:15:25.6866571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6867037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6867605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6868139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6868729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6869166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6869985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6870461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6870923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6871417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6872170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6872895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6873427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6873893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6874401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t9coarx
2022-12-01T11:15:25.6874953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t9coarx/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6875479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpclr75dyq
2022-12-01T11:15:25.6876033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpclr75dyq/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6876556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6877049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6877523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6878008Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6878359Z ok (5.656s)
2022-12-01T11:15:25.6878511Z 
2022-12-01T11:15:25.6878763Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6879093Z Ran 1 test in 5.656s
2022-12-01T11:15:25.6879254Z 
2022-12-01T11:15:25.6879345Z OK
2022-12-01T11:15:25.6879478Z 
2022-12-01T11:15:25.6879602Z Generating XML reports...
2022-12-01T11:15:25.6880210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104915.xml
2022-12-01T11:15:25.6881029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6881491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6882060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6882544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6882777Z 
2022-12-01T11:15:25.6882884Z Running tests...
2022-12-01T11:15:25.6883289Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6883807Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6884410Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6884999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64493
2022-12-01T11:15:25.6885445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64494
2022-12-01T11:15:25.6886061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6886518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6887098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6887558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6888147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6888601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6889186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6889699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6890170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6890678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6891330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6892037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6892570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6893055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6893555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81zoo531
2022-12-01T11:15:25.6894106Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81zoo531/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6894645Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0ax4km_
2022-12-01T11:15:25.6895191Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0ax4km_/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6895694Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6896189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6896543Z ok (5.967s)
2022-12-01T11:15:25.6896691Z 
2022-12-01T11:15:25.6896949Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6897276Z Ran 1 test in 5.967s
2022-12-01T11:15:25.6897440Z 
2022-12-01T11:15:25.6897531Z OK
2022-12-01T11:15:25.6897666Z 
2022-12-01T11:15:25.6897831Z Generating XML reports...
2022-12-01T11:15:25.6898463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104923.xml
2022-12-01T11:15:25.6899191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6899649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6900211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6900689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6900923Z 
2022-12-01T11:15:25.6901031Z Running tests...
2022-12-01T11:15:25.6901437Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6901953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6902558Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6903131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64836
2022-12-01T11:15:25.6903572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64837
2022-12-01T11:15:25.6904190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6904642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6905226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6905692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6906278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6906738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6907345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6907831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6908293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6908802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6909724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6910445Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6910978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6911568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6912066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcm9anvsn
2022-12-01T11:15:25.6912616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcm9anvsn/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6913166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpww6rfi2s
2022-12-01T11:15:25.6913701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpww6rfi2s/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6914229Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6914726Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6915211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6915683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6916132Z ok (6.666s)
2022-12-01T11:15:25.6916286Z 
2022-12-01T11:15:25.6916559Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6916869Z Ran 1 test in 6.667s
2022-12-01T11:15:25.6917032Z 
2022-12-01T11:15:25.6917123Z OK
2022-12-01T11:15:25.6917257Z 
2022-12-01T11:15:25.6917381Z Generating XML reports...
2022-12-01T11:15:25.6917998Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104931.xml
2022-12-01T11:15:25.6918708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6919169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6919752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6920232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6920453Z 
2022-12-01T11:15:25.6920561Z Running tests...
2022-12-01T11:15:25.6920970Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6921508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6922065Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6922613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65185
2022-12-01T11:15:25.6923072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65186
2022-12-01T11:15:25.6923689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6924134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6924715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6925264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6925852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6926306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6926880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6927350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6927793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6928300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6928971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6929684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6930204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6930686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6931197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpskzt62q0
2022-12-01T11:15:25.6931733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpskzt62q0/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6932278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd82rt4vj
2022-12-01T11:15:25.6932822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd82rt4vj/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6933345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6933893Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6934384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6934875Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6935209Z ok (6.340s)
2022-12-01T11:15:25.6935361Z 
2022-12-01T11:15:25.6935640Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6935968Z Ran 1 test in 6.340s
2022-12-01T11:15:25.6936132Z 
2022-12-01T11:15:25.6936224Z OK
2022-12-01T11:15:25.6936341Z 
2022-12-01T11:15:25.6936467Z Generating XML reports...
2022-12-01T11:15:25.6937088Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104940.xml
2022-12-01T11:15:25.6937814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6938262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6938849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6939327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6939563Z 
2022-12-01T11:15:25.6939670Z Running tests...
2022-12-01T11:15:25.6940056Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6940590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6941188Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6941763Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65531
2022-12-01T11:15:25.6942205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65532
2022-12-01T11:15:25.6942872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6943344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6943894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6944352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6944932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6945410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6945995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6946471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6946939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6947431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6948103Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6948805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6949334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6950022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6950532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppk6zh_ji
2022-12-01T11:15:25.6951086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppk6zh_ji/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6951724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptipjrio4
2022-12-01T11:15:25.6952256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptipjrio4/_remote_module_non_scriptable.py
2022-12-01T11:15:25.6952780Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6953271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6953745Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6954233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.6954585Z ok (5.813s)
2022-12-01T11:15:25.6954734Z 
2022-12-01T11:15:25.6955015Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6955326Z Ran 1 test in 5.813s
2022-12-01T11:15:25.6955487Z 
2022-12-01T11:15:25.6955585Z OK
2022-12-01T11:15:25.6955720Z 
2022-12-01T11:15:25.6955847Z Generating XML reports...
2022-12-01T11:15:25.6956457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104948.xml
2022-12-01T11:15:25.6957191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6957650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6958236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6958705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6958941Z 
2022-12-01T11:15:25.6959051Z Running tests...
2022-12-01T11:15:25.6959460Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6959978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6960617Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6961723Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.656s)
2022-12-01T11:15:25.6962263Z 
2022-12-01T11:15:25.6962529Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6962858Z Ran 1 test in 1.656s
2022-12-01T11:15:25.6963005Z 
2022-12-01T11:15:25.6963113Z OK (skipped=1)
2022-12-01T11:15:25.6963268Z 
2022-12-01T11:15:25.6963393Z Generating XML reports...
2022-12-01T11:15:25.6964010Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104956.xml
2022-12-01T11:15:25.6964732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6965192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6965777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6966259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6966474Z 
2022-12-01T11:15:25.6966581Z Running tests...
2022-12-01T11:15:25.6966987Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6967517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6968112Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6968652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65932
2022-12-01T11:15:25.6969177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65933
2022-12-01T11:15:25.6969805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6970249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6970836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6971314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6971885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6972340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6972917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6973393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6973849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6974364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6975031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6975735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6976253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6976736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6977086Z ok (4.074s)
2022-12-01T11:15:25.6977235Z 
2022-12-01T11:15:25.6977495Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6977827Z Ran 1 test in 4.074s
2022-12-01T11:15:25.6977988Z 
2022-12-01T11:15:25.6978126Z OK
2022-12-01T11:15:25.6978268Z 
2022-12-01T11:15:25.6978393Z Generating XML reports...
2022-12-01T11:15:25.6978995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105000.xml
2022-12-01T11:15:25.6979719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6980179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6980745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6981226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6981457Z 
2022-12-01T11:15:25.6981566Z Running tests...
2022-12-01T11:15:25.6981973Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6982505Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6983077Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6984165Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.612s)
2022-12-01T11:15:25.6984696Z 
2022-12-01T11:15:25.6984963Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6985273Z Ran 1 test in 1.612s
2022-12-01T11:15:25.6985433Z 
2022-12-01T11:15:25.6985539Z OK (skipped=1)
2022-12-01T11:15:25.6985693Z 
2022-12-01T11:15:25.6985817Z Generating XML reports...
2022-12-01T11:15:25.6986415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105007.xml
2022-12-01T11:15:25.6987231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6987689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6988273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6988736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6988970Z 
2022-12-01T11:15:25.6989078Z Running tests...
2022-12-01T11:15:25.6989783Z ----------------------------------------------------------------------
2022-12-01T11:15:25.6990337Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.6990864Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.6991388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66320
2022-12-01T11:15:25.6991848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66321
2022-12-01T11:15:25.6992455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6992917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6993502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6993979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6994548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.6994999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.6995577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.6996116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.6996591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.6997099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.6997771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6998457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.6998997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.6999482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.6999832Z ok (5.591s)
2022-12-01T11:15:25.6999972Z 
2022-12-01T11:15:25.7000240Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7000571Z Ran 1 test in 5.591s
2022-12-01T11:15:25.7000734Z 
2022-12-01T11:15:25.7000826Z OK
2022-12-01T11:15:25.7000958Z 
2022-12-01T11:15:25.7001064Z Generating XML reports...
2022-12-01T11:15:25.7001685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105010.xml
2022-12-01T11:15:25.7002411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7002869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7003438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7003914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7004147Z 
2022-12-01T11:15:25.7004256Z Running tests...
2022-12-01T11:15:25.7004727Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7005267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7005780Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2022-12-01T11:15:25.7006071Z 
2022-12-01T11:15:25.7006334Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7006644Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7006805Z 
2022-12-01T11:15:25.7006912Z OK (skipped=1)
2022-12-01T11:15:25.7007066Z 
2022-12-01T11:15:25.7007190Z Generating XML reports...
2022-12-01T11:15:25.7007786Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105018.xml
2022-12-01T11:15:25.7008518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7008978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7009564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7010024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7010256Z 
2022-12-01T11:15:25.7010362Z Running tests...
2022-12-01T11:15:25.7010767Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7011281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7011750Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.7012235Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7012724Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66747
2022-12-01T11:15:25.7013163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66748
2022-12-01T11:15:25.7013831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7014297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7014884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7015349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7015929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7016379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7016946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7017419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7017886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7018399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7019050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7019749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7020283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7020765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7021264Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprb0e3n9i
2022-12-01T11:15:25.7021802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp40rmjiuc
2022-12-01T11:15:25.7022461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprb0e3n9i/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7023004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp40rmjiuc/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7023388Z ok (3.908s)
2022-12-01T11:15:25.7023541Z 
2022-12-01T11:15:25.7023817Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7024143Z Ran 1 test in 3.908s
2022-12-01T11:15:25.7024288Z 
2022-12-01T11:15:25.7024383Z OK
2022-12-01T11:15:25.7024519Z 
2022-12-01T11:15:25.7024644Z Generating XML reports...
2022-12-01T11:15:25.7025264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105021.xml
2022-12-01T11:15:25.7025979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7026439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7027032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7027513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7027729Z 
2022-12-01T11:15:25.7027838Z Running tests...
2022-12-01T11:15:25.7028244Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7028780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7029255Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.7029968Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7030462Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67021
2022-12-01T11:15:25.7030923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67022
2022-12-01T11:15:25.7031615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7032090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7032680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7033145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7033732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7034186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7034766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7035226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7035689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7036206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7036885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7037573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7038103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7038615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp130xf5o1
2022-12-01T11:15:25.7039149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp130xf5o1/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7039666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7040258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3s2rra4
2022-12-01T11:15:25.7040801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3s2rra4/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7041307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.7041804Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.7042162Z ok (4.170s)
2022-12-01T11:15:25.7042313Z 
2022-12-01T11:15:25.7042573Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7042903Z Ran 1 test in 4.170s
2022-12-01T11:15:25.7043067Z 
2022-12-01T11:15:25.7043158Z OK
2022-12-01T11:15:25.7043292Z 
2022-12-01T11:15:25.7043415Z Generating XML reports...
2022-12-01T11:15:25.7044018Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105027.xml
2022-12-01T11:15:25.7044759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7045219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7045785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7046267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7046501Z 
2022-12-01T11:15:25.7046608Z Running tests...
2022-12-01T11:15:25.7047014Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7047535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7048040Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.7048590Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7049158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67340
2022-12-01T11:15:25.7049610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67341
2022-12-01T11:15:25.7050228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7050686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7051254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7051729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7052316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7052766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7053327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7053812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7054275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7054765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7055438Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7056143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7056677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7057142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7057715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppay3wi4h
2022-12-01T11:15:25.7058254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9yub7h3
2022-12-01T11:15:25.7058801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppay3wi4h/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7059339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9yub7h3/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7059860Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.7060356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.7060697Z ok (3.925s)
2022-12-01T11:15:25.7060844Z 
2022-12-01T11:15:25.7061121Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7061453Z Ran 1 test in 3.925s
2022-12-01T11:15:25.7061615Z 
2022-12-01T11:15:25.7061707Z OK
2022-12-01T11:15:25.7061823Z 
2022-12-01T11:15:25.7061953Z Generating XML reports...
2022-12-01T11:15:25.7062573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105033.xml
2022-12-01T11:15:25.7063307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7063749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7064335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7064814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7065048Z 
2022-12-01T11:15:25.7065156Z Running tests...
2022-12-01T11:15:25.7065540Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7066076Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7066559Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2022-12-01T11:15:25.7067098Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7067600Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67609
2022-12-01T11:15:25.7068105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67610
2022-12-01T11:15:25.7068731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7069170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7069991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7070451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7071016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7071502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7072102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7072580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7073022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7073529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7074195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7074903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7075420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7076030Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdbsothl
2022-12-01T11:15:25.7076587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdbsothl/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7077085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7077592Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknpieryp
2022-12-01T11:15:25.7078137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknpieryp/_remote_module_non_scriptable.py
2022-12-01T11:15:25.7078523Z ok (4.122s)
2022-12-01T11:15:25.7078678Z 
2022-12-01T11:15:25.7078938Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7079269Z Ran 1 test in 4.122s
2022-12-01T11:15:25.7079431Z 
2022-12-01T11:15:25.7079523Z OK
2022-12-01T11:15:25.7079657Z 
2022-12-01T11:15:25.7079762Z Generating XML reports...
2022-12-01T11:15:25.7080390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105039.xml
2022-12-01T11:15:25.7081121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7081579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7082143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7082615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7082851Z 
2022-12-01T11:15:25.7082959Z Running tests...
2022-12-01T11:15:25.7083365Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7083886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7084389Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7084950Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67878
2022-12-01T11:15:25.7085407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67879
2022-12-01T11:15:25.7086029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7086487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7087069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7087534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7088120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7088578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7089150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7089627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7090087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7090600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7091251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7091951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7092485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7092966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7093606Z STAGE:2022-12-01 10:50:49 67878:67878 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7094188Z STAGE:2022-12-01 10:50:49 67879:67879 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7094780Z STAGE:2022-12-01 10:50:49 67878:67878 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7095347Z STAGE:2022-12-01 10:50:49 67879:67879 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7095926Z STAGE:2022-12-01 10:50:49 67878:67878 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7096509Z STAGE:2022-12-01 10:50:49 67879:67879 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7097096Z STAGE:2022-12-01 10:50:49 67878:67878 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7097664Z STAGE:2022-12-01 10:50:49 67879:67879 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7098027Z ok (3.896s)
2022-12-01T11:15:25.7098175Z 
2022-12-01T11:15:25.7098446Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7098758Z Ran 1 test in 3.896s
2022-12-01T11:15:25.7098923Z 
2022-12-01T11:15:25.7099014Z OK
2022-12-01T11:15:25.7099148Z 
2022-12-01T11:15:25.7099271Z Generating XML reports...
2022-12-01T11:15:25.7099888Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105046.xml
2022-12-01T11:15:25.7100601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7101058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7101641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7102118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7102339Z 
2022-12-01T11:15:25.7102447Z Running tests...
2022-12-01T11:15:25.7102898Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7103448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7103965Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7104479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68191
2022-12-01T11:15:25.7104938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68192
2022-12-01T11:15:25.7105561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7106007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7106395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7106599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7106970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7107143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7107526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7107716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7107950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7108198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7108605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7109085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7109319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7109921Z STAGE:2022-12-01 10:50:55 68192:68192 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7110678Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7110793Z   warnings.warn(
2022-12-01T11:15:25.7111027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7111346Z STAGE:2022-12-01 10:50:55 68191:68191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7112110Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7112226Z   warnings.warn(
2022-12-01T11:15:25.7112569Z STAGE:2022-12-01 10:50:55 68191:68191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7112903Z STAGE:2022-12-01 10:50:55 68192:68192 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7113232Z STAGE:2022-12-01 10:50:55 68191:68191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7113552Z STAGE:2022-12-01 10:50:55 68192:68192 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7113887Z STAGE:2022-12-01 10:50:55 68191:68191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7114223Z STAGE:2022-12-01 10:50:55 68192:68192 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7114619Z STAGE:2022-12-01 10:50:55 68192:68192 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7114950Z STAGE:2022-12-01 10:50:55 68191:68191 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7115286Z STAGE:2022-12-01 10:50:55 68192:68192 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7115613Z STAGE:2022-12-01 10:50:55 68191:68191 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7115714Z ok (4.043s)
2022-12-01T11:15:25.7115735Z 
2022-12-01T11:15:25.7115997Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7116109Z Ran 1 test in 4.043s
2022-12-01T11:15:25.7116129Z 
2022-12-01T11:15:25.7116220Z OK
2022-12-01T11:15:25.7116239Z 
2022-12-01T11:15:25.7116363Z Generating XML reports...
2022-12-01T11:15:25.7116817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105052.xml
2022-12-01T11:15:25.7117203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7117380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7117768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7117961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7117981Z 
2022-12-01T11:15:25.7118090Z Running tests...
2022-12-01T11:15:25.7118353Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7118675Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7118962Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7119258Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68457
2022-12-01T11:15:25.7119481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68458
2022-12-01T11:15:25.7119863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7120038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7120422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7120614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7120984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7121157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7121516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7121716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7121969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7122216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7122623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7123023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7123256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7123487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7123734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7124040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7124456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7124856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7125400Z STAGE:2022-12-01 10:51:02 68458:68458 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:51:02 68457:68457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7125422Z 
2022-12-01T11:15:25.7126171Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7126291Z   warnings.warn(
2022-12-01T11:15:25.7127038Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7127150Z   warnings.warn(
2022-12-01T11:15:25.7127500Z STAGE:2022-12-01 10:51:02 68458:68458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7127832Z STAGE:2022-12-01 10:51:02 68457:68457 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7128371Z STAGE:2022-12-01 10:51:02 68458:68458 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:51:02 68457:68457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7128438Z 
2022-12-01T11:15:25.7129001Z STAGE:2022-12-01 10:51:02 68457:68457 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:51:02 68458:68458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7129022Z 
2022-12-01T11:15:25.7129336Z STAGE:2022-12-01 10:51:02 68458:68458 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7129666Z STAGE:2022-12-01 10:51:02 68457:68457 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7130001Z STAGE:2022-12-01 10:51:02 68458:68458 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7130330Z STAGE:2022-12-01 10:51:02 68457:68457 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7130433Z ok (4.009s)
2022-12-01T11:15:25.7130453Z 
2022-12-01T11:15:25.7130721Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7130834Z Ran 1 test in 4.010s
2022-12-01T11:15:25.7130857Z 
2022-12-01T11:15:25.7130949Z OK
2022-12-01T11:15:25.7130969Z 
2022-12-01T11:15:25.7131097Z Generating XML reports...
2022-12-01T11:15:25.7131543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105058.xml
2022-12-01T11:15:25.7131922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7132101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7132490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7132686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7132706Z 
2022-12-01T11:15:25.7132812Z Running tests...
2022-12-01T11:15:25.7133077Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7133395Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7133708Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7133941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68723
2022-12-01T11:15:25.7134163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68724
2022-12-01T11:15:25.7134542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7134722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7135107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7135300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7135667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7135848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7136212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7136404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7136656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7136905Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7137313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7137715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7137950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7138240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7138404Z skip: Skipped due to small world size. (3.836s)
2022-12-01T11:15:25.7138424Z 
2022-12-01T11:15:25.7138676Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7138787Z Ran 1 test in 3.836s
2022-12-01T11:15:25.7138807Z 
2022-12-01T11:15:25.7138914Z OK (skipped=1)
2022-12-01T11:15:25.7138932Z 
2022-12-01T11:15:25.7139056Z Generating XML reports...
2022-12-01T11:15:25.7139519Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105104.xml
2022-12-01T11:15:25.7139900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7140079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7140465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7140649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7140685Z 
2022-12-01T11:15:25.7140776Z Running tests...
2022-12-01T11:15:25.7141040Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7141361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7141644Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7141865Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69000
2022-12-01T11:15:25.7142086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69001
2022-12-01T11:15:25.7142467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7142646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7143067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7143270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7143644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7143817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7144201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7144393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7144643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7144893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7145306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7145693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7145929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7146268Z STAGE:2022-12-01 10:51:14 69001:69001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7147023Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7147136Z   warnings.warn(
2022-12-01T11:15:25.7147372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7147770Z STAGE:2022-12-01 10:51:14 69000:69000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7148522Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7148634Z   warnings.warn(
2022-12-01T11:15:25.7148959Z STAGE:2022-12-01 10:51:14 69000:69000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7149295Z STAGE:2022-12-01 10:51:14 69001:69001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7149884Z STAGE:2022-12-01 10:51:14 69000:69000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7150207Z STAGE:2022-12-01 10:51:14 69001:69001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7150543Z STAGE:2022-12-01 10:51:14 69000:69000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7150879Z STAGE:2022-12-01 10:51:14 69001:69001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7151209Z STAGE:2022-12-01 10:51:14 69000:69000 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7151533Z STAGE:2022-12-01 10:51:14 69001:69001 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7151867Z STAGE:2022-12-01 10:51:14 69000:69000 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7152178Z STAGE:2022-12-01 10:51:14 69001:69001 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7152282Z ok (4.143s)
2022-12-01T11:15:25.7152303Z 
2022-12-01T11:15:25.7152568Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7152681Z Ran 1 test in 4.144s
2022-12-01T11:15:25.7152704Z 
2022-12-01T11:15:25.7152795Z OK
2022-12-01T11:15:25.7152814Z 
2022-12-01T11:15:25.7153012Z Generating XML reports...
2022-12-01T11:15:25.7153485Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105110.xml
2022-12-01T11:15:25.7153863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7154042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7154413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7154608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7154628Z 
2022-12-01T11:15:25.7154735Z Running tests...
2022-12-01T11:15:25.7154998Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7155317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7155609Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7155831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69331
2022-12-01T11:15:25.7156050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69332
2022-12-01T11:15:25.7156409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7156588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7156972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7157167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7157536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7157779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7158167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7158360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7158609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7158841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7159250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7159654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7159888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7160128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7160468Z STAGE:2022-12-01 10:51:20 69331:69331 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7160796Z STAGE:2022-12-01 10:51:20 69332:69332 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7161544Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7161658Z   warnings.warn(
2022-12-01T11:15:25.7162403Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2441: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7162502Z   warnings.warn(
2022-12-01T11:15:25.7162893Z STAGE:2022-12-01 10:51:20 69331:69331 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7163240Z STAGE:2022-12-01 10:51:20 69332:69332 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7163341Z ok (4.100s)
2022-12-01T11:15:25.7163360Z 
2022-12-01T11:15:25.7163626Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7163738Z Ran 1 test in 4.101s
2022-12-01T11:15:25.7163758Z 
2022-12-01T11:15:25.7163849Z OK
2022-12-01T11:15:25.7163868Z 
2022-12-01T11:15:25.7163994Z Generating XML reports...
2022-12-01T11:15:25.7164436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105117.xml
2022-12-01T11:15:25.7164813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7164995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7165389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7165583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7165603Z 
2022-12-01T11:15:25.7165710Z Running tests...
2022-12-01T11:15:25.7165974Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7166293Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7166559Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7166764Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69594
2022-12-01T11:15:25.7166985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69595
2022-12-01T11:15:25.7167426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7167604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7167988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7168231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7168606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7168783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7169145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7169337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7169591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7169846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7170250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7170654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7170888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7171229Z STAGE:2022-12-01 10:51:27 69595:69595 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7171462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7171781Z STAGE:2022-12-01 10:51:27 69594:69594 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7172120Z STAGE:2022-12-01 10:51:27 69594:69594 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7172508Z STAGE:2022-12-01 10:51:27 69595:69595 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7172850Z STAGE:2022-12-01 10:51:27 69594:69594 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7173172Z STAGE:2022-12-01 10:51:27 69595:69595 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7173501Z STAGE:2022-12-01 10:51:27 69594:69594 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7173828Z STAGE:2022-12-01 10:51:27 69595:69595 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7173930Z ok (4.093s)
2022-12-01T11:15:25.7173950Z 
2022-12-01T11:15:25.7174214Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7174313Z Ran 1 test in 4.093s
2022-12-01T11:15:25.7174332Z 
2022-12-01T11:15:25.7174428Z OK
2022-12-01T11:15:25.7174448Z 
2022-12-01T11:15:25.7174574Z Generating XML reports...
2022-12-01T11:15:25.7175045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105123.xml
2022-12-01T11:15:25.7175425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7175602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7175991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7176188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7176208Z 
2022-12-01T11:15:25.7176298Z Running tests...
2022-12-01T11:15:25.7176561Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7176877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7177221Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T11:15:25.7177242Z 
2022-12-01T11:15:25.7177503Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7177616Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7177636Z 
2022-12-01T11:15:25.7177745Z OK (skipped=1)
2022-12-01T11:15:25.7177764Z 
2022-12-01T11:15:25.7177887Z Generating XML reports...
2022-12-01T11:15:25.7178343Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105130.xml
2022-12-01T11:15:25.7178704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7178881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7179265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7179463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7179485Z 
2022-12-01T11:15:25.7179593Z Running tests...
2022-12-01T11:15:25.7179857Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7180179Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7180460Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2022-12-01T11:15:25.7180480Z 
2022-12-01T11:15:25.7180749Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7180843Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7180862Z 
2022-12-01T11:15:25.7180968Z OK (skipped=1)
2022-12-01T11:15:25.7180988Z 
2022-12-01T11:15:25.7181111Z Generating XML reports...
2022-12-01T11:15:25.7181564Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105132.xml
2022-12-01T11:15:25.7181993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7182177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7182565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7182757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7182777Z 
2022-12-01T11:15:25.7182884Z Running tests...
2022-12-01T11:15:25.7183128Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7183446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7183713Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7183934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70091
2022-12-01T11:15:25.7184162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70092
2022-12-01T11:15:25.7184541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7184718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7185107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7185284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7185655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7185834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7186210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7186456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7186711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7186963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7187373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7187777Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7187992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7188224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7188468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7188722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7189126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7189716Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7190071Z STAGE:2022-12-01 10:51:37 70091:70091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7190396Z STAGE:2022-12-01 10:51:37 70092:70092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7190949Z STAGE:2022-12-01 10:51:37 70091:70091 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:51:37 70092:70092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7190971Z 
2022-12-01T11:15:25.7191286Z STAGE:2022-12-01 10:51:37 70092:70092 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7191690Z STAGE:2022-12-01 10:51:37 70091:70091 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7192257Z STAGE:2022-12-01 10:51:37 70091:70091 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:51:37 70092:70092 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7192278Z 
2022-12-01T11:15:25.7192380Z ok (3.862s)
2022-12-01T11:15:25.7192399Z 
2022-12-01T11:15:25.7192662Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7192777Z Ran 1 test in 3.863s
2022-12-01T11:15:25.7192796Z 
2022-12-01T11:15:25.7192887Z OK
2022-12-01T11:15:25.7192906Z 
2022-12-01T11:15:25.7193032Z Generating XML reports...
2022-12-01T11:15:25.7193489Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105134.xml
2022-12-01T11:15:25.7193867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7194036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7194426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7194623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7194643Z 
2022-12-01T11:15:25.7194751Z Running tests...
2022-12-01T11:15:25.7195019Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7195338Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7195603Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7195824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70416
2022-12-01T11:15:25.7196028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70417
2022-12-01T11:15:25.7196483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7196662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7197048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7197241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7197609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7197783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7198162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7198351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7198590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7198837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7199246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7199653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7199889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7200121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7200280Z skip: Skipped due to small world size. (3.946s)
2022-12-01T11:15:25.7200300Z 
2022-12-01T11:15:25.7200567Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7200667Z Ran 1 test in 3.946s
2022-12-01T11:15:25.7200703Z 
2022-12-01T11:15:25.7200838Z OK (skipped=1)
2022-12-01T11:15:25.7200859Z 
2022-12-01T11:15:25.7200990Z Generating XML reports...
2022-12-01T11:15:25.7201451Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105140.xml
2022-12-01T11:15:25.7201828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7202005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7202390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7202584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7202603Z 
2022-12-01T11:15:25.7202711Z Running tests...
2022-12-01T11:15:25.7202959Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7203284Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7203588Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T11:15:25.7203608Z 
2022-12-01T11:15:25.7203870Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7203982Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7204002Z 
2022-12-01T11:15:25.7204109Z OK (skipped=1)
2022-12-01T11:15:25.7204127Z 
2022-12-01T11:15:25.7204250Z Generating XML reports...
2022-12-01T11:15:25.7204708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105146.xml
2022-12-01T11:15:25.7205080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7205239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7205686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7205879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7205898Z 
2022-12-01T11:15:25.7206005Z Running tests...
2022-12-01T11:15:25.7206267Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7206584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7206890Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2022-12-01T11:15:25.7206911Z 
2022-12-01T11:15:25.7207169Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7207280Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7207299Z 
2022-12-01T11:15:25.7207388Z OK (skipped=1)
2022-12-01T11:15:25.7207407Z 
2022-12-01T11:15:25.7207535Z Generating XML reports...
2022-12-01T11:15:25.7207994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105148.xml
2022-12-01T11:15:25.7208371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7208548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7208932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7209125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7209145Z 
2022-12-01T11:15:25.7209252Z Running tests...
2022-12-01T11:15:25.7209499Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7209818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7210108Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T11:15:25.7210176Z 
2022-12-01T11:15:25.7210447Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7210560Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7210579Z 
2022-12-01T11:15:25.7210685Z OK (skipped=1)
2022-12-01T11:15:25.7210704Z 
2022-12-01T11:15:25.7210827Z Generating XML reports...
2022-12-01T11:15:25.7211281Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105150.xml
2022-12-01T11:15:25.7211658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7211818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7212205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7212407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7212426Z 
2022-12-01T11:15:25.7212537Z Running tests...
2022-12-01T11:15:25.7212802Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7213121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7213427Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2022-12-01T11:15:25.7213447Z 
2022-12-01T11:15:25.7213709Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7213823Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7213843Z 
2022-12-01T11:15:25.7213931Z OK (skipped=1)
2022-12-01T11:15:25.7213966Z 
2022-12-01T11:15:25.7214072Z Generating XML reports...
2022-12-01T11:15:25.7214527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105152.xml
2022-12-01T11:15:25.7214965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7215141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7215520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7215713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7215732Z 
2022-12-01T11:15:25.7215840Z Running tests...
2022-12-01T11:15:25.7216104Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7216405Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7216685Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7216908Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71022
2022-12-01T11:15:25.7217138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71023
2022-12-01T11:15:25.7217516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7217692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7218075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7218267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7218621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7218796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7219180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7219376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7219671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7219926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7220332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7220733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7220966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7221182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7221282Z ok (4.256s)
2022-12-01T11:15:25.7221302Z 
2022-12-01T11:15:25.7221568Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7221685Z Ran 1 test in 4.256s
2022-12-01T11:15:25.7221705Z 
2022-12-01T11:15:25.7221800Z OK
2022-12-01T11:15:25.7221819Z 
2022-12-01T11:15:25.7221943Z Generating XML reports...
2022-12-01T11:15:25.7222405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105154.xml
2022-12-01T11:15:25.7222782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7222961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7223329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7223520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7223539Z 
2022-12-01T11:15:25.7223646Z Running tests...
2022-12-01T11:15:25.7223907Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7224287Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7224563Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7224784Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71301
2022-12-01T11:15:25.7225003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71302
2022-12-01T11:15:25.7225365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7225540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7225920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7226112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7226479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7226661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7227039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7227230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7227480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7227710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7228119Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7228523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7228758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7229034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7229285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7229772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7230192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7230588Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7230816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.7231061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.7231475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.7231872Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.7232118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2022-12-01T11:15:25.7232359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2022-12-01T11:15:25.7232753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T11:15:25.7233152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2022-12-01T11:15:25.7233255Z ok (4.219s)
2022-12-01T11:15:25.7233275Z 
2022-12-01T11:15:25.7233524Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7233727Z Ran 1 test in 4.219s
2022-12-01T11:15:25.7233749Z 
2022-12-01T11:15:25.7233842Z OK
2022-12-01T11:15:25.7233862Z 
2022-12-01T11:15:25.7233985Z Generating XML reports...
2022-12-01T11:15:25.7234449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105201.xml
2022-12-01T11:15:25.7234825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7235001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7235387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7235564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7235600Z 
2022-12-01T11:15:25.7235691Z Running tests...
2022-12-01T11:15:25.7235956Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7236282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7236551Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.003s)
2022-12-01T11:15:25.7236570Z 
2022-12-01T11:15:25.7236832Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7236944Z Ran 1 test in 0.003s
2022-12-01T11:15:25.7236963Z 
2022-12-01T11:15:25.7237069Z OK (skipped=1)
2022-12-01T11:15:25.7237088Z 
2022-12-01T11:15:25.7237209Z Generating XML reports...
2022-12-01T11:15:25.7237648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105207.xml
2022-12-01T11:15:25.7238027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7238204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7238652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7238854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7238873Z 
2022-12-01T11:15:25.7238981Z Running tests...
2022-12-01T11:15:25.7239249Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7239566Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7239857Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7240062Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71707
2022-12-01T11:15:25.7240285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71708
2022-12-01T11:15:25.7240660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7240846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7241236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7241430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7241799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7241974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7242337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7242529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7242777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7243096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7243504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7243905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7244138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7244372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7244616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7244845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7245247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7245654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7245988Z STAGE:2022-12-01 10:52:13 71707:71707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7246313Z STAGE:2022-12-01 10:52:13 71708:71708 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7247067Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7247185Z   warnings.warn(
2022-12-01T11:15:25.7247933Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7248090Z   warnings.warn(
2022-12-01T11:15:25.7248446Z STAGE:2022-12-01 10:52:13 71707:71707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7248760Z STAGE:2022-12-01 10:52:13 71708:71708 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7249092Z STAGE:2022-12-01 10:52:13 71707:71707 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7249419Z STAGE:2022-12-01 10:52:13 71708:71708 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7249751Z STAGE:2022-12-01 10:52:13 71707:71707 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7250089Z STAGE:2022-12-01 10:52:13 71708:71708 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7250191Z ok (4.038s)
2022-12-01T11:15:25.7250211Z 
2022-12-01T11:15:25.7250482Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7250598Z Ran 1 test in 4.038s
2022-12-01T11:15:25.7250618Z 
2022-12-01T11:15:25.7250710Z OK
2022-12-01T11:15:25.7250729Z 
2022-12-01T11:15:25.7250836Z Generating XML reports...
2022-12-01T11:15:25.7251293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105209.xml
2022-12-01T11:15:25.7251671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7251847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7252233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7252424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7252444Z 
2022-12-01T11:15:25.7252551Z Running tests...
2022-12-01T11:15:25.7252873Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7253178Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7253468Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7253692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72035
2022-12-01T11:15:25.7253913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72036
2022-12-01T11:15:25.7254290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7254468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7254841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7255017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7255413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7255591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7255975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7256169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7256417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7256666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7257070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7257476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7257757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7257995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7258223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7258468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7258875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7259274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7259819Z STAGE:2022-12-01 10:52:19 72036:72036 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:52:19 72035:72035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7259843Z 
2022-12-01T11:15:25.7260604Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7260718Z   warnings.warn(
2022-12-01T11:15:25.7261467Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7261577Z   warnings.warn(
2022-12-01T11:15:25.7261922Z STAGE:2022-12-01 10:52:19 72035:72035 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7262240Z STAGE:2022-12-01 10:52:19 72036:72036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7262638Z STAGE:2022-12-01 10:52:19 72035:72035 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7262962Z STAGE:2022-12-01 10:52:19 72036:72036 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7263298Z STAGE:2022-12-01 10:52:19 72036:72036 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7263627Z STAGE:2022-12-01 10:52:19 72035:72035 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7263728Z ok (3.956s)
2022-12-01T11:15:25.7263748Z 
2022-12-01T11:15:25.7264013Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7264126Z Ran 1 test in 3.956s
2022-12-01T11:15:25.7264145Z 
2022-12-01T11:15:25.7264237Z OK
2022-12-01T11:15:25.7264257Z 
2022-12-01T11:15:25.7264365Z Generating XML reports...
2022-12-01T11:15:25.7264830Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105216.xml
2022-12-01T11:15:25.7265219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7265397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7265781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7265976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7265995Z 
2022-12-01T11:15:25.7266103Z Running tests...
2022-12-01T11:15:25.7266368Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7266667Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7266966Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7267193Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72289
2022-12-01T11:15:25.7267458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72290
2022-12-01T11:15:25.7267843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7268020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7268453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7268645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7269010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7269167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7269734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7269939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7270191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7270439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7270854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7271255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7271492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7271723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7271949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7272313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7272719Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7273116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7273453Z STAGE:2022-12-01 10:52:26 72289:72289 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7274202Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7274315Z   warnings.warn(
2022-12-01T11:15:25.7274660Z STAGE:2022-12-01 10:52:26 72290:72290 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7275409Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7275519Z   warnings.warn(
2022-12-01T11:15:25.7275844Z STAGE:2022-12-01 10:52:26 72289:72289 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7276178Z STAGE:2022-12-01 10:52:26 72290:72290 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7276715Z STAGE:2022-12-01 10:52:26 72289:72289 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:52:26 72290:72290 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7276736Z 
2022-12-01T11:15:25.7277342Z STAGE:2022-12-01 10:52:26 72289:72289 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:52:26 72290:72290 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7277368Z 
2022-12-01T11:15:25.7277475Z ok (4.134s)
2022-12-01T11:15:25.7277495Z 
2022-12-01T11:15:25.7277769Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7277883Z Ran 1 test in 4.134s
2022-12-01T11:15:25.7277902Z 
2022-12-01T11:15:25.7277994Z OK
2022-12-01T11:15:25.7278013Z 
2022-12-01T11:15:25.7278137Z Generating XML reports...
2022-12-01T11:15:25.7278594Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105222.xml
2022-12-01T11:15:25.7278957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7279134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7279519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7279721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7279742Z 
2022-12-01T11:15:25.7279849Z Running tests...
2022-12-01T11:15:25.7280115Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7280436Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7280728Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7280933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72616
2022-12-01T11:15:25.7281155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72617
2022-12-01T11:15:25.7281532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7281768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7282160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7282353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7282717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7282891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7283269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7283444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7283692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7283940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7284350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7284755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7284988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7285221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7285464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7285711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7286101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7286557Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7286908Z STAGE:2022-12-01 10:52:32 72616:72616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7287655Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7287769Z   warnings.warn(
2022-12-01T11:15:25.7288105Z STAGE:2022-12-01 10:52:32 72617:72617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7288855Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7288970Z   warnings.warn(
2022-12-01T11:15:25.7289536Z STAGE:2022-12-01 10:52:32 72617:72617 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:52:32 72616:72616 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7289557Z 
2022-12-01T11:15:25.7290095Z STAGE:2022-12-01 10:52:32 72617:72617 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:52:32 72616:72616 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7290115Z 
2022-12-01T11:15:25.7290456Z STAGE:2022-12-01 10:52:32 72616:72616 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7290768Z STAGE:2022-12-01 10:52:32 72617:72617 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7290872Z ok (4.046s)
2022-12-01T11:15:25.7290891Z 
2022-12-01T11:15:25.7291155Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7291318Z Ran 1 test in 4.047s
2022-12-01T11:15:25.7291337Z 
2022-12-01T11:15:25.7291432Z OK
2022-12-01T11:15:25.7291450Z 
2022-12-01T11:15:25.7291575Z Generating XML reports...
2022-12-01T11:15:25.7292036Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105228.xml
2022-12-01T11:15:25.7292415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7292591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7292958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7293150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7293170Z 
2022-12-01T11:15:25.7293276Z Running tests...
2022-12-01T11:15:25.7293540Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7293869Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7294157Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7294382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72856
2022-12-01T11:15:25.7294603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72857
2022-12-01T11:15:25.7294966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7295145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7295533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7295726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7296142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7296324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7296707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7296898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7297149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7297382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7297788Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7298187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7298427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7298659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7298820Z skip: Skipped due to small world size. (3.821s)
2022-12-01T11:15:25.7298840Z 
2022-12-01T11:15:25.7299108Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7299220Z Ran 1 test in 3.822s
2022-12-01T11:15:25.7299239Z 
2022-12-01T11:15:25.7299346Z OK (skipped=1)
2022-12-01T11:15:25.7299366Z 
2022-12-01T11:15:25.7299472Z Generating XML reports...
2022-12-01T11:15:25.7299934Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105235.xml
2022-12-01T11:15:25.7300314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7300492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7300944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7301139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7301159Z 
2022-12-01T11:15:25.7301266Z Running tests...
2022-12-01T11:15:25.7301533Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7301835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7302121Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7302342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73147
2022-12-01T11:15:25.7302562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73148
2022-12-01T11:15:25.7302938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7303123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7303511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7303706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7304071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7304229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7304612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7304805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7305055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7305365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7305789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7306194Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7306427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7306659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7306801Z skip: Skipped due to small world size. (3.797s)
2022-12-01T11:15:25.7306820Z 
2022-12-01T11:15:25.7307087Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7307199Z Ran 1 test in 3.798s
2022-12-01T11:15:25.7307218Z 
2022-12-01T11:15:25.7307324Z OK (skipped=1)
2022-12-01T11:15:25.7307349Z 
2022-12-01T11:15:25.7307471Z Generating XML reports...
2022-12-01T11:15:25.7307933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105241.xml
2022-12-01T11:15:25.7308307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7308483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7308850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7309042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7309062Z 
2022-12-01T11:15:25.7309169Z Running tests...
2022-12-01T11:15:25.7309668Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7310004Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7310399Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7310620Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73423
2022-12-01T11:15:25.7310841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73424
2022-12-01T11:15:25.7311219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7311379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7311770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7311963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7312333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7312508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7312896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7313089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7313336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7313564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7313970Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7314372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7314604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7314833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7315064Z skip: Skipped due to small world size. (3.843s)
2022-12-01T11:15:25.7315086Z 
2022-12-01T11:15:25.7315361Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7315472Z Ran 1 test in 3.843s
2022-12-01T11:15:25.7315492Z 
2022-12-01T11:15:25.7315597Z OK (skipped=1)
2022-12-01T11:15:25.7315616Z 
2022-12-01T11:15:25.7315722Z Generating XML reports...
2022-12-01T11:15:25.7316180Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105247.xml
2022-12-01T11:15:25.7316557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7316736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7317120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7317318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7317341Z 
2022-12-01T11:15:25.7317448Z Running tests...
2022-12-01T11:15:25.7317711Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7318030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7318294Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7318515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73680
2022-12-01T11:15:25.7318735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73681
2022-12-01T11:15:25.7319114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7319291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7319739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7319932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7320301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7320460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7320839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7321028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7321277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7321525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7321930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7322396Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7322630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7322860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7323002Z skip: Skipped due to small world size. (3.757s)
2022-12-01T11:15:25.7323022Z 
2022-12-01T11:15:25.7323290Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7323401Z Ran 1 test in 3.758s
2022-12-01T11:15:25.7323420Z 
2022-12-01T11:15:25.7323524Z OK (skipped=1)
2022-12-01T11:15:25.7323543Z 
2022-12-01T11:15:25.7323666Z Generating XML reports...
2022-12-01T11:15:25.7324125Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105253.xml
2022-12-01T11:15:25.7324554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7324738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7325124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7325299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7325319Z 
2022-12-01T11:15:25.7325425Z Running tests...
2022-12-01T11:15:25.7325688Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7326005Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7326280Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7326503Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73991
2022-12-01T11:15:25.7326729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73992
2022-12-01T11:15:25.7327107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7327266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7327648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7327839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7328209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7328381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7328765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7329013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7329264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7329512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7329903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7330300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7330536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7330768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7331112Z STAGE:2022-12-01 10:53:03 73992:73992 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7331450Z STAGE:2022-12-01 10:53:03 73991:73991 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7332202Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7332315Z   warnings.warn(
2022-12-01T11:15:25.7333064Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7333172Z   warnings.warn(
2022-12-01T11:15:25.7333711Z STAGE:2022-12-01 10:53:03 73992:73992 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:53:03 73991:73991 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7333797Z 
2022-12-01T11:15:25.7334122Z STAGE:2022-12-01 10:53:03 73992:73992 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7334443Z STAGE:2022-12-01 10:53:03 73991:73991 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7334778Z STAGE:2022-12-01 10:53:03 73992:73992 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7335108Z STAGE:2022-12-01 10:53:03 73991:73991 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7335214Z ok (3.971s)
2022-12-01T11:15:25.7335233Z 
2022-12-01T11:15:25.7335496Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7335609Z Ran 1 test in 3.971s
2022-12-01T11:15:25.7335629Z 
2022-12-01T11:15:25.7335720Z OK
2022-12-01T11:15:25.7335738Z 
2022-12-01T11:15:25.7335844Z Generating XML reports...
2022-12-01T11:15:25.7336312Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105259.xml
2022-12-01T11:15:25.7336692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7336866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7337250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7337443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7337463Z 
2022-12-01T11:15:25.7337569Z Running tests...
2022-12-01T11:15:25.7337833Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7338132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7338440Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7338723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74306
2022-12-01T11:15:25.7338944Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74307
2022-12-01T11:15:25.7339322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7339499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7339884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7340077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7340446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7340604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7340989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7341179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7341430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7341680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7342084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7342484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7342718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7342948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7343730Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7343850Z   warnings.warn(
2022-12-01T11:15:25.7344599Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7344709Z   warnings.warn(
2022-12-01T11:15:25.7344810Z ok (3.915s)
2022-12-01T11:15:25.7344830Z 
2022-12-01T11:15:25.7345096Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7345206Z Ran 1 test in 3.915s
2022-12-01T11:15:25.7345226Z 
2022-12-01T11:15:25.7345315Z OK
2022-12-01T11:15:25.7345339Z 
2022-12-01T11:15:25.7345464Z Generating XML reports...
2022-12-01T11:15:25.7345907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105305.xml
2022-12-01T11:15:25.7346287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7346465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7346846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7347040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7347059Z 
2022-12-01T11:15:25.7347166Z Running tests...
2022-12-01T11:15:25.7347428Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7347747Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7348080Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7348286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74617
2022-12-01T11:15:25.7348504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74618
2022-12-01T11:15:25.7348879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7349058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7349616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7349809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7350184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7350354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7350729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7350918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7351165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7351412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7351818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7352220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7352453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7352794Z STAGE:2022-12-01 10:53:15 74618:74618 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7353617Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7353738Z   warnings.warn(
2022-12-01T11:15:25.7353952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7354292Z STAGE:2022-12-01 10:53:15 74617:74617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7355037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7355152Z   warnings.warn(
2022-12-01T11:15:25.7355496Z STAGE:2022-12-01 10:53:15 74617:74617 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7355827Z STAGE:2022-12-01 10:53:15 74618:74618 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7356152Z STAGE:2022-12-01 10:53:15 74617:74617 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7356473Z STAGE:2022-12-01 10:53:15 74618:74618 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7356804Z STAGE:2022-12-01 10:53:15 74618:74618 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7357125Z STAGE:2022-12-01 10:53:15 74617:74617 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7357226Z ok (4.022s)
2022-12-01T11:15:25.7357246Z 
2022-12-01T11:15:25.7357510Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7357623Z Ran 1 test in 4.022s
2022-12-01T11:15:25.7357702Z 
2022-12-01T11:15:25.7357795Z OK
2022-12-01T11:15:25.7357814Z 
2022-12-01T11:15:25.7357941Z Generating XML reports...
2022-12-01T11:15:25.7358408Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105311.xml
2022-12-01T11:15:25.7358785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7358948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7359335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7359531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7359550Z 
2022-12-01T11:15:25.7359661Z Running tests...
2022-12-01T11:15:25.7359925Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7360240Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7360526Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7360747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74932
2022-12-01T11:15:25.7360965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74933
2022-12-01T11:15:25.7361325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7361502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7361884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7362077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7362450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7362674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7363070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7363259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7363491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7363741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7364150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7364550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7364783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7365124Z STAGE:2022-12-01 10:53:21 74933:74933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7365874Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7365982Z   warnings.warn(
2022-12-01T11:15:25.7366215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7366553Z STAGE:2022-12-01 10:53:21 74932:74932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7367282Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7367451Z   warnings.warn(
2022-12-01T11:15:25.7367797Z STAGE:2022-12-01 10:53:21 74932:74932 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7368174Z STAGE:2022-12-01 10:53:21 74933:74933 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7368506Z STAGE:2022-12-01 10:53:21 74932:74932 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7368829Z STAGE:2022-12-01 10:53:21 74933:74933 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7369154Z STAGE:2022-12-01 10:53:21 74932:74932 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7369486Z STAGE:2022-12-01 10:53:21 74933:74933 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7369583Z ok (3.883s)
2022-12-01T11:15:25.7369603Z 
2022-12-01T11:15:25.7369852Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7369964Z Ran 1 test in 3.883s
2022-12-01T11:15:25.7369986Z 
2022-12-01T11:15:25.7370076Z OK
2022-12-01T11:15:25.7370094Z 
2022-12-01T11:15:25.7370217Z Generating XML reports...
2022-12-01T11:15:25.7370673Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105318.xml
2022-12-01T11:15:25.7371043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7371221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7371604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7371782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7371816Z 
2022-12-01T11:15:25.7371908Z Running tests...
2022-12-01T11:15:25.7372162Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7372531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7372807Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7373030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75227
2022-12-01T11:15:25.7373249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75228
2022-12-01T11:15:25.7373628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7373804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7374168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7374356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7374723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7374893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7375272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7375458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7375702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7375944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7376348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7376734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7377039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7377269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7377611Z STAGE:2022-12-01 10:53:27 75228:75228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7377933Z STAGE:2022-12-01 10:53:27 75227:75227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7378685Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7378797Z   warnings.warn(
2022-12-01T11:15:25.7379539Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1577: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2022-12-01T11:15:25.7379648Z   warnings.warn(
2022-12-01T11:15:25.7380200Z STAGE:2022-12-01 10:53:27 75228:75228 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:53:27 75227:75227 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7380221Z 
2022-12-01T11:15:25.7380535Z STAGE:2022-12-01 10:53:27 75228:75228 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7380860Z STAGE:2022-12-01 10:53:27 75227:75227 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7381194Z STAGE:2022-12-01 10:53:27 75228:75228 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7381521Z STAGE:2022-12-01 10:53:27 75227:75227 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7381622Z ok (3.902s)
2022-12-01T11:15:25.7381641Z 
2022-12-01T11:15:25.7381948Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7382065Z Ran 1 test in 3.902s
2022-12-01T11:15:25.7382085Z 
2022-12-01T11:15:25.7382173Z OK
2022-12-01T11:15:25.7382192Z 
2022-12-01T11:15:25.7382312Z Generating XML reports...
2022-12-01T11:15:25.7382762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105324.xml
2022-12-01T11:15:25.7383136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7383308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7383689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7383884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7383903Z 
2022-12-01T11:15:25.7384012Z Running tests...
2022-12-01T11:15:25.7384277Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7384590Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7384883Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7385108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75538
2022-12-01T11:15:25.7385329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75539
2022-12-01T11:15:25.7385690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7385864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7386246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7386489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7386864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7387038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7387414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7387599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7387831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7388078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7388479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7388883Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7389122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7389569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7389684Z ok (3.995s)
2022-12-01T11:15:25.7389706Z 
2022-12-01T11:15:25.7389979Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7390090Z Ran 1 test in 3.995s
2022-12-01T11:15:25.7390110Z 
2022-12-01T11:15:25.7390184Z OK
2022-12-01T11:15:25.7390203Z 
2022-12-01T11:15:25.7390321Z Generating XML reports...
2022-12-01T11:15:25.7390781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105330.xml
2022-12-01T11:15:25.7391153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7391332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7391787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7391989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7392008Z 
2022-12-01T11:15:25.7392112Z Running tests...
2022-12-01T11:15:25.7392374Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7392676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7392952Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7393172Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75774
2022-12-01T11:15:25.7393391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75775
2022-12-01T11:15:25.7393773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7393946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7394325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7394515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7394865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7395037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7395410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7395596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7395842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7396163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7396577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7396982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7397216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7397431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7397672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7397916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7398323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7398728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7399071Z STAGE:2022-12-01 10:53:40 75775:75775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7399399Z STAGE:2022-12-01 10:53:40 75774:75774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7399736Z STAGE:2022-12-01 10:53:40 75774:75774 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7400061Z STAGE:2022-12-01 10:53:40 75775:75775 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7400582Z STAGE:2022-12-01 10:53:40 75774:75774 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:53:40 75775:75775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7400625Z 
2022-12-01T11:15:25.7400995Z STAGE:2022-12-01 10:53:40 75775:75775 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7401339Z STAGE:2022-12-01 10:53:40 75774:75774 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7401441Z ok (4.180s)
2022-12-01T11:15:25.7401461Z 
2022-12-01T11:15:25.7401723Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7401835Z Ran 1 test in 4.180s
2022-12-01T11:15:25.7401857Z 
2022-12-01T11:15:25.7401942Z OK
2022-12-01T11:15:25.7401962Z 
2022-12-01T11:15:25.7402089Z Generating XML reports...
2022-12-01T11:15:25.7402697Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105336.xml
2022-12-01T11:15:25.7403437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7403895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7404492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7404955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7405191Z 
2022-12-01T11:15:25.7405300Z Running tests...
2022-12-01T11:15:25.7405702Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7406229Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7406740Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7407251Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76066
2022-12-01T11:15:25.7407709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76067
2022-12-01T11:15:25.7408307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7408830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7409422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7409904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7410472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7410936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7411513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7411987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7412432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7412944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7413614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7414340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7414845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7415315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7415792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7416269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7416922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7417663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7418304Z STAGE:2022-12-01 10:53:46 76067:76067 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7418861Z STAGE:2022-12-01 10:53:46 76066:76066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7419437Z STAGE:2022-12-01 10:53:46 76067:76067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7420013Z STAGE:2022-12-01 10:53:46 76066:76066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7420809Z STAGE:2022-12-01 10:53:46 76067:76067 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:53:46 76066:76066 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7421198Z 
2022-12-01T11:15:25.7421536Z STAGE:2022-12-01 10:53:46 76066:76066 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7422098Z STAGE:2022-12-01 10:53:46 76067:76067 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7422442Z ok (4.089s)
2022-12-01T11:15:25.7422588Z 
2022-12-01T11:15:25.7422854Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7423183Z Ran 1 test in 4.089s
2022-12-01T11:15:25.7423325Z 
2022-12-01T11:15:25.7423417Z OK
2022-12-01T11:15:25.7423550Z 
2022-12-01T11:15:25.7423674Z Generating XML reports...
2022-12-01T11:15:25.7424284Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105343.xml
2022-12-01T11:15:25.7424987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7425439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7426013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7426547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7426759Z 
2022-12-01T11:15:25.7426869Z Running tests...
2022-12-01T11:15:25.7427276Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7427804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7428307Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7428813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76399
2022-12-01T11:15:25.7429259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76400
2022-12-01T11:15:25.7430054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7430498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7431099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7431572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7432131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7432587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7433169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7433651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7434100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7434580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7435352Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7436059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7436577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7437030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7437507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7437997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7438649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7439325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7439955Z STAGE:2022-12-01 10:53:53 76400:76400 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7440526Z STAGE:2022-12-01 10:53:53 76399:76399 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7441086Z STAGE:2022-12-01 10:53:53 76399:76399 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7441681Z STAGE:2022-12-01 10:53:53 76399:76399 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7442280Z STAGE:2022-12-01 10:53:53 76400:76400 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7442845Z STAGE:2022-12-01 10:53:53 76400:76400 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7443414Z STAGE:2022-12-01 10:53:53 76399:76399 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7444063Z STAGE:2022-12-01 10:53:53 76400:76400 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7444413Z ok (4.292s)
2022-12-01T11:15:25.7444559Z 
2022-12-01T11:15:25.7444827Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7445139Z Ran 1 test in 4.292s
2022-12-01T11:15:25.7445302Z 
2022-12-01T11:15:25.7445392Z OK
2022-12-01T11:15:25.7445527Z 
2022-12-01T11:15:25.7445648Z Generating XML reports...
2022-12-01T11:15:25.7446254Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105349.xml
2022-12-01T11:15:25.7446952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7447401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7447973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7448441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7448668Z 
2022-12-01T11:15:25.7448777Z Running tests...
2022-12-01T11:15:25.7449190Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7449750Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7450252Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7450776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76670
2022-12-01T11:15:25.7451232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76671
2022-12-01T11:15:25.7451835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7452272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7452921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7453414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7454000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7454437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7455025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7455498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7455955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7456445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7457118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7457809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7458334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7458793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7459274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7459774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7460413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7461098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7461784Z STAGE:2022-12-01 10:53:59 76671:76671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7462354Z STAGE:2022-12-01 10:53:59 76670:76670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7462920Z STAGE:2022-12-01 10:53:59 76670:76670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7463507Z STAGE:2022-12-01 10:53:59 76671:76671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7464082Z STAGE:2022-12-01 10:53:59 76670:76670 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7464648Z STAGE:2022-12-01 10:53:59 76671:76671 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7465207Z STAGE:2022-12-01 10:53:59 76671:76671 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7465790Z STAGE:2022-12-01 10:53:59 76670:76670 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7466139Z ok (4.134s)
2022-12-01T11:15:25.7466293Z 
2022-12-01T11:15:25.7466545Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7466874Z Ran 1 test in 4.135s
2022-12-01T11:15:25.7467037Z 
2022-12-01T11:15:25.7467130Z OK
2022-12-01T11:15:25.7467262Z 
2022-12-01T11:15:25.7467387Z Generating XML reports...
2022-12-01T11:15:25.7467981Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105356.xml
2022-12-01T11:15:25.7468755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7469215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7470121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7470600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7470838Z 
2022-12-01T11:15:25.7471030Z Running tests...
2022-12-01T11:15:25.7471448Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7471962Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7472479Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7472976Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76955
2022-12-01T11:15:25.7473414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76956
2022-12-01T11:15:25.7474026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7474480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7475062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7475530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7476115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7476625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7477184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7477656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7478112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7478619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7479266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7480064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7480590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7481071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7481461Z skip: Skipped due to small world size. (3.984s)
2022-12-01T11:15:25.7481662Z 
2022-12-01T11:15:25.7481928Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7482256Z Ran 1 test in 3.984s
2022-12-01T11:15:25.7482418Z 
2022-12-01T11:15:25.7482508Z OK (skipped=1)
2022-12-01T11:15:25.7482661Z 
2022-12-01T11:15:25.7482785Z Generating XML reports...
2022-12-01T11:15:25.7483402Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105402.xml
2022-12-01T11:15:25.7484127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7484562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7485142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7485616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7485847Z 
2022-12-01T11:15:25.7485955Z Running tests...
2022-12-01T11:15:25.7486342Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7486878Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7487389Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7487868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77231
2022-12-01T11:15:25.7488332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77232
2022-12-01T11:15:25.7489000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7489460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7490012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7490465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7491040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7491496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7492092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7492566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7493030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7493515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7494180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7494875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7495404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7495865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7496262Z skip: Skipped due to small world size. (4.099s)
2022-12-01T11:15:25.7496466Z 
2022-12-01T11:15:25.7496736Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7497107Z Ran 1 test in 4.099s
2022-12-01T11:15:25.7497273Z 
2022-12-01T11:15:25.7497383Z OK (skipped=1)
2022-12-01T11:15:25.7497537Z 
2022-12-01T11:15:25.7497661Z Generating XML reports...
2022-12-01T11:15:25.7498276Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105408.xml
2022-12-01T11:15:25.7498979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7499433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7500015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7500472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7500707Z 
2022-12-01T11:15:25.7500811Z Running tests...
2022-12-01T11:15:25.7501215Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7501754Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7502263Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7502771Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77540
2022-12-01T11:15:25.7503224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77541
2022-12-01T11:15:25.7503822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7504277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7504850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7505323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7505938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7506396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7506973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7507423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7507882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7508385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7509046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7509921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7510467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7510947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7511354Z skip: Skipped due to small world size. (3.869s)
2022-12-01T11:15:25.7511538Z 
2022-12-01T11:15:25.7511806Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7512135Z Ran 1 test in 3.869s
2022-12-01T11:15:25.7512294Z 
2022-12-01T11:15:25.7512402Z OK (skipped=1)
2022-12-01T11:15:25.7512554Z 
2022-12-01T11:15:25.7512660Z Generating XML reports...
2022-12-01T11:15:25.7513272Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105415.xml
2022-12-01T11:15:25.7513996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7514451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7515114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7515590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7515822Z 
2022-12-01T11:15:25.7515931Z Running tests...
2022-12-01T11:15:25.7516316Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7516847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7517363Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7517857Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77775
2022-12-01T11:15:25.7518295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77776
2022-12-01T11:15:25.7518903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7519365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7519945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7520395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7520979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7521423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7521978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7522450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7522910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7523485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7524147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7524845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7525372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7525853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7526239Z skip: Skipped due to small world size. (3.991s)
2022-12-01T11:15:25.7526442Z 
2022-12-01T11:15:25.7526714Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7527040Z Ran 1 test in 3.992s
2022-12-01T11:15:25.7527200Z 
2022-12-01T11:15:25.7527290Z OK (skipped=1)
2022-12-01T11:15:25.7527448Z 
2022-12-01T11:15:25.7527574Z Generating XML reports...
2022-12-01T11:15:25.7528185Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105421.xml
2022-12-01T11:15:25.7528908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7529345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7529923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7530397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7530629Z 
2022-12-01T11:15:25.7530720Z Running tests...
2022-12-01T11:15:25.7531123Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7531653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7532216Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7532690Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78088
2022-12-01T11:15:25.7533147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78089
2022-12-01T11:15:25.7533759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7534193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7534763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7535213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7535787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7536245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7536846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7537317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7537759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7538267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7538937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7539632Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7540137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7540721Z STAGE:2022-12-01 10:54:30 78089:78089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7541252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7541843Z STAGE:2022-12-01 10:54:30 78088:78088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7542413Z STAGE:2022-12-01 10:54:30 78088:78088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7542996Z STAGE:2022-12-01 10:54:30 78089:78089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7543567Z STAGE:2022-12-01 10:54:30 78088:78088 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7544135Z STAGE:2022-12-01 10:54:30 78089:78089 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7544699Z STAGE:2022-12-01 10:54:30 78089:78089 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7545281Z STAGE:2022-12-01 10:54:30 78088:78088 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7545636Z ok (3.936s)
2022-12-01T11:15:25.7545785Z 
2022-12-01T11:15:25.7546035Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7546363Z Ran 1 test in 3.936s
2022-12-01T11:15:25.7546523Z 
2022-12-01T11:15:25.7546618Z OK
2022-12-01T11:15:25.7546752Z 
2022-12-01T11:15:25.7546857Z Generating XML reports...
2022-12-01T11:15:25.7547469Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105427.xml
2022-12-01T11:15:25.7548194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7548650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7549216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7549988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7550222Z 
2022-12-01T11:15:25.7550339Z Running tests...
2022-12-01T11:15:25.7550756Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7551276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7551778Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7552265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78352
2022-12-01T11:15:25.7552706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78353
2022-12-01T11:15:25.7553317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7553773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7554355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7554822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7555405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7555854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7556414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7556886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7557339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7557843Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7558491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7559267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7559804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7560281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7560855Z STAGE:2022-12-01 10:54:37 78353:78353 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7561423Z STAGE:2022-12-01 10:54:37 78352:78352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7562002Z STAGE:2022-12-01 10:54:37 78352:78352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7562565Z STAGE:2022-12-01 10:54:37 78353:78353 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7563345Z STAGE:2022-12-01 10:54:37 78353:78353 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:54:37 78352:78352 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7563732Z 
2022-12-01T11:15:25.7564069Z STAGE:2022-12-01 10:54:37 78352:78352 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7564653Z STAGE:2022-12-01 10:54:37 78353:78353 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7564978Z ok (4.014s)
2022-12-01T11:15:25.7565128Z 
2022-12-01T11:15:25.7565398Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7565727Z Ran 1 test in 4.014s
2022-12-01T11:15:25.7565889Z 
2022-12-01T11:15:25.7565982Z OK
2022-12-01T11:15:25.7566098Z 
2022-12-01T11:15:25.7566219Z Generating XML reports...
2022-12-01T11:15:25.7566825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105433.xml
2022-12-01T11:15:25.7567548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7568090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7568729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7569205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7569434Z 
2022-12-01T11:15:25.7569543Z Running tests...
2022-12-01T11:15:25.7569927Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7570464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7570980Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7571458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78672
2022-12-01T11:15:25.7571915Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78673
2022-12-01T11:15:25.7572534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7572990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7573552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7574028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7591314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7591773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7592364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7592835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7593413Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7593938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7594602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7595293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7595800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7596271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7596850Z STAGE:2022-12-01 10:54:45 78673:78673 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7597879Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:15:25.7598518Z   warnings.warn(
2022-12-01T11:15:25.7598996Z STAGE:2022-12-01 10:54:45 78672:78672 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7599993Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:15:25.7600631Z   warnings.warn(
2022-12-01T11:15:25.7601091Z STAGE:2022-12-01 10:54:45 78672:78672 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7601675Z STAGE:2022-12-01 10:54:45 78673:78673 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7602337Z STAGE:2022-12-01 10:54:45 78672:78672 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7602906Z STAGE:2022-12-01 10:54:45 78673:78673 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7603464Z STAGE:2022-12-01 10:54:45 78672:78672 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7604036Z STAGE:2022-12-01 10:54:45 78673:78673 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7604376Z ok (6.071s)
2022-12-01T11:15:25.7604526Z 
2022-12-01T11:15:25.7604777Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7605110Z Ran 1 test in 6.071s
2022-12-01T11:15:25.7605272Z 
2022-12-01T11:15:25.7605364Z OK
2022-12-01T11:15:25.7605497Z 
2022-12-01T11:15:25.7605620Z Generating XML reports...
2022-12-01T11:15:25.7606215Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105439.xml
2022-12-01T11:15:25.7606943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7607395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7607955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7608434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7608664Z 
2022-12-01T11:15:25.7608771Z Running tests...
2022-12-01T11:15:25.7609177Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7609694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7610213Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7610718Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78996
2022-12-01T11:15:25.7611208Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78997
2022-12-01T11:15:25.7611833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7612280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7612852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7613307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7613887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7614331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7614899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7615355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7615812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7616313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7616957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7617654Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7618179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7618654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7619220Z STAGE:2022-12-01 10:54:53 78997:78997 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7620330Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:15:25.7620980Z   warnings.warn(
2022-12-01T11:15:25.7621457Z STAGE:2022-12-01 10:54:53 78996:78996 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7622506Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1445: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:15:25.7623138Z   warnings.warn(
2022-12-01T11:15:25.7623625Z STAGE:2022-12-01 10:54:53 78996:78996 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7624228Z STAGE:2022-12-01 10:54:53 78997:78997 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7624789Z STAGE:2022-12-01 10:54:53 78996:78996 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7625360Z STAGE:2022-12-01 10:54:53 78997:78997 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7625937Z STAGE:2022-12-01 10:54:53 78996:78996 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7626511Z STAGE:2022-12-01 10:54:53 78997:78997 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7626839Z ok (5.912s)
2022-12-01T11:15:25.7626988Z 
2022-12-01T11:15:25.7627257Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7627590Z Ran 1 test in 5.912s
2022-12-01T11:15:25.7627752Z 
2022-12-01T11:15:25.7627827Z OK
2022-12-01T11:15:25.7627960Z 
2022-12-01T11:15:25.7628087Z Generating XML reports...
2022-12-01T11:15:25.7628759Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105448.xml
2022-12-01T11:15:25.7629752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7630198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7630791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7631263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7631493Z 
2022-12-01T11:15:25.7631601Z Running tests...
2022-12-01T11:15:25.7631987Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7632525Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7633036Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7633523Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79412
2022-12-01T11:15:25.7633979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79413
2022-12-01T11:15:25.7634589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7635041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7635603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7636078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7636653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7637090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7637779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7638253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7638714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7639195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7639855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7640550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7641074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7641640Z STAGE:2022-12-01 10:54:59 79413:79413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7642129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7642707Z STAGE:2022-12-01 10:54:59 79412:79412 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7643295Z STAGE:2022-12-01 10:54:59 79412:79412 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7643865Z STAGE:2022-12-01 10:54:59 79413:79413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7644444Z STAGE:2022-12-01 10:54:59 79412:79412 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7645010Z STAGE:2022-12-01 10:54:59 79413:79413 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7645577Z STAGE:2022-12-01 10:54:59 79412:79412 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7646158Z STAGE:2022-12-01 10:54:59 79413:79413 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7646502Z ok (3.928s)
2022-12-01T11:15:25.7646653Z 
2022-12-01T11:15:25.7646990Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7647314Z Ran 1 test in 3.929s
2022-12-01T11:15:25.7647479Z 
2022-12-01T11:15:25.7647573Z OK
2022-12-01T11:15:25.7647706Z 
2022-12-01T11:15:25.7647836Z Generating XML reports...
2022-12-01T11:15:25.7648432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105456.xml
2022-12-01T11:15:25.7649153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7649608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7650184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7650642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7650877Z 
2022-12-01T11:15:25.7650990Z Running tests...
2022-12-01T11:15:25.7651407Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7651943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7652439Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7652941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79660
2022-12-01T11:15:25.7653394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79661
2022-12-01T11:15:25.7653987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7654442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7655020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7655556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7656122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7656574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7657146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7657601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7658053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7658555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7659213Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7659896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7660422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7660895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7661237Z ok (5.793s)
2022-12-01T11:15:25.7661366Z 
2022-12-01T11:15:25.7661639Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7661971Z Ran 1 test in 5.793s
2022-12-01T11:15:25.7662131Z 
2022-12-01T11:15:25.7662224Z OK
2022-12-01T11:15:25.7662360Z 
2022-12-01T11:15:25.7662465Z Generating XML reports...
2022-12-01T11:15:25.7663072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105502.xml
2022-12-01T11:15:25.7663788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7664297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7664864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7665338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7665570Z 
2022-12-01T11:15:25.7665680Z Running tests...
2022-12-01T11:15:25.7666066Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7666636Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7667145Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7667629Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80023
2022-12-01T11:15:25.7668061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80024
2022-12-01T11:15:25.7668733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7669189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7670064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7670520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7671096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7671546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7672101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7672573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7673030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7673635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7674274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7674970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7675496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7676083Z STAGE:2022-12-01 10:55:13 80024:80024 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7676537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7677113Z STAGE:2022-12-01 10:55:13 80023:80023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7677705Z STAGE:2022-12-01 10:55:13 80023:80023 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7678293Z STAGE:2022-12-01 10:55:13 80024:80024 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7678604Z STAGE:2022-12-01 10:55:13 80023:80023 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7678923Z STAGE:2022-12-01 10:55:13 80024:80024 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7679253Z STAGE:2022-12-01 10:55:13 80024:80024 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7679592Z STAGE:2022-12-01 10:55:13 80023:80023 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7679694Z ok (3.916s)
2022-12-01T11:15:25.7679714Z 
2022-12-01T11:15:25.7679981Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7680095Z Ran 1 test in 3.917s
2022-12-01T11:15:25.7680114Z 
2022-12-01T11:15:25.7680216Z OK
2022-12-01T11:15:25.7680236Z 
2022-12-01T11:15:25.7680341Z Generating XML reports...
2022-12-01T11:15:25.7680864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105510.xml
2022-12-01T11:15:25.7681259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7681443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7681828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7682020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7682039Z 
2022-12-01T11:15:25.7682148Z Running tests...
2022-12-01T11:15:25.7682415Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7682730Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7682982Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7683207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80302
2022-12-01T11:15:25.7683425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80303
2022-12-01T11:15:25.7683798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7683975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7684360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7684551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7684921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7685150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7685517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7685713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7685964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7686208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7686609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7687008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7687237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7687587Z STAGE:2022-12-01 10:55:19 80303:80303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7687821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7688137Z STAGE:2022-12-01 10:55:20 80302:80302 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7688479Z STAGE:2022-12-01 10:55:20 80302:80302 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7688812Z STAGE:2022-12-01 10:55:20 80303:80303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7689143Z STAGE:2022-12-01 10:55:20 80302:80302 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7689464Z STAGE:2022-12-01 10:55:20 80303:80303 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7689795Z STAGE:2022-12-01 10:55:20 80303:80303 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7690172Z STAGE:2022-12-01 10:55:20 80302:80302 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7690280Z ok (3.911s)
2022-12-01T11:15:25.7690300Z 
2022-12-01T11:15:25.7690566Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7690661Z Ran 1 test in 3.911s
2022-12-01T11:15:25.7690680Z 
2022-12-01T11:15:25.7690775Z OK
2022-12-01T11:15:25.7690794Z 
2022-12-01T11:15:25.7690917Z Generating XML reports...
2022-12-01T11:15:25.7691376Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105516.xml
2022-12-01T11:15:25.7691753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7691931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7692320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7692515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7692538Z 
2022-12-01T11:15:25.7692629Z Running tests...
2022-12-01T11:15:25.7692893Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7693207Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7693475Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7693695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80594
2022-12-01T11:15:25.7693917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80595
2022-12-01T11:15:25.7694290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7694466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7694911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7695085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7695451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7695626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7696001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7696191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7696441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7696688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7697090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7697502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7697716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7697943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7698284Z STAGE:2022-12-01 10:55:26 80594:80594 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7698615Z STAGE:2022-12-01 10:55:26 80595:80595 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7698952Z STAGE:2022-12-01 10:55:26 80594:80594 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7699289Z STAGE:2022-12-01 10:55:26 80595:80595 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7699874Z STAGE:2022-12-01 10:55:26 80595:80595 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:55:26 80594:80594 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7699895Z 
2022-12-01T11:15:25.7700243Z STAGE:2022-12-01 10:55:26 80595:80595 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7700567Z STAGE:2022-12-01 10:55:26 80594:80594 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7700652Z ok (3.976s)
2022-12-01T11:15:25.7700692Z 
2022-12-01T11:15:25.7700941Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7701057Z Ran 1 test in 3.976s
2022-12-01T11:15:25.7701076Z 
2022-12-01T11:15:25.7701169Z OK
2022-12-01T11:15:25.7701188Z 
2022-12-01T11:15:25.7701313Z Generating XML reports...
2022-12-01T11:15:25.7701771Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105522.xml
2022-12-01T11:15:25.7702150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7702330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7702712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7702887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7702925Z 
2022-12-01T11:15:25.7703016Z Running tests...
2022-12-01T11:15:25.7703285Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7703602Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7703864Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7704083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80890
2022-12-01T11:15:25.7704402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80891
2022-12-01T11:15:25.7704778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7704957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7705315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7705509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7705879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7706056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7706434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7706629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7706880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7707129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7707514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7707922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7708151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7708381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7708721Z STAGE:2022-12-01 10:55:34 80891:80891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7709052Z STAGE:2022-12-01 10:55:34 80890:80890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7709701Z STAGE:2022-12-01 10:55:34 80890:80890 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7710073Z STAGE:2022-12-01 10:55:34 80891:80891 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7710407Z STAGE:2022-12-01 10:55:34 80891:80891 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7710710Z STAGE:2022-12-01 10:55:34 80890:80890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7711256Z STAGE:2022-12-01 10:55:35 80890:80890 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:55:35 80891:80891 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7711279Z 
2022-12-01T11:15:25.7711811Z STAGE:2022-12-01 10:55:35 80891:80891 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:55:35 80890:80890 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7711838Z 
2022-12-01T11:15:25.7712174Z STAGE:2022-12-01 10:55:35 80890:80890 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7712504Z STAGE:2022-12-01 10:55:35 80891:80891 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7712607Z ok (7.750s)
2022-12-01T11:15:25.7712626Z 
2022-12-01T11:15:25.7712895Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7713009Z Ran 1 test in 7.751s
2022-12-01T11:15:25.7713028Z 
2022-12-01T11:15:25.7713121Z OK
2022-12-01T11:15:25.7713140Z 
2022-12-01T11:15:25.7713266Z Generating XML reports...
2022-12-01T11:15:25.7713719Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105528.xml
2022-12-01T11:15:25.7714076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7714342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7714725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7714917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7714937Z 
2022-12-01T11:15:25.7715049Z Running tests...
2022-12-01T11:15:25.7715316Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7715631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7715906Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7716109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81258
2022-12-01T11:15:25.7716331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81259
2022-12-01T11:15:25.7716714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7716892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7717274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7717465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7717828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7718003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7718379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7718549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7718797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7719093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7719502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7719897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7720131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7720363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7720704Z STAGE:2022-12-01 10:55:44 81259:81259 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7721026Z STAGE:2022-12-01 10:55:44 81258:81258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7721353Z STAGE:2022-12-01 10:55:44 81258:81258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7721691Z STAGE:2022-12-01 10:55:44 81259:81259 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7722017Z STAGE:2022-12-01 10:55:44 81258:81258 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7722344Z STAGE:2022-12-01 10:55:44 81259:81259 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7722889Z STAGE:2022-12-01 10:55:45 81258:81258 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:55:45 81259:81259 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7722909Z 
2022-12-01T11:15:25.7723438Z STAGE:2022-12-01 10:55:45 81258:81258 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 10:55:45 81259:81259 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7723517Z 
2022-12-01T11:15:25.7723861Z STAGE:2022-12-01 10:55:45 81258:81258 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7724193Z STAGE:2022-12-01 10:55:45 81259:81259 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7724295Z ok (7.919s)
2022-12-01T11:15:25.7724314Z 
2022-12-01T11:15:25.7724582Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7724677Z Ran 1 test in 7.919s
2022-12-01T11:15:25.7724712Z 
2022-12-01T11:15:25.7724785Z OK
2022-12-01T11:15:25.7724803Z 
2022-12-01T11:15:25.7724924Z Generating XML reports...
2022-12-01T11:15:25.7725381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105538.xml
2022-12-01T11:15:25.7725752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7725926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7726316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7726509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7726528Z 
2022-12-01T11:15:25.7726633Z Running tests...
2022-12-01T11:15:25.7726875Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7727192Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7727461Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7727683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81725
2022-12-01T11:15:25.7727900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81726
2022-12-01T11:15:25.7728274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7728497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7728893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7729084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7729433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7729613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7729982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7730169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7730415Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7730663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7731072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7731473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7731699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7731912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7732251Z STAGE:2022-12-01 10:55:54 81725:81725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7732575Z STAGE:2022-12-01 10:55:54 81726:81726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7733121Z STAGE:2022-12-01 10:55:54 81725:81725 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:55:54 81726:81726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7733195Z 
2022-12-01T11:15:25.7733537Z STAGE:2022-12-01 10:55:54 81726:81726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7733865Z STAGE:2022-12-01 10:55:54 81725:81725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7734408Z STAGE:2022-12-01 10:55:55 81726:81726 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:55:55 81725:81725 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7734428Z 
2022-12-01T11:15:25.7734757Z STAGE:2022-12-01 10:55:55 81726:81726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7735087Z STAGE:2022-12-01 10:55:55 81725:81725 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7735420Z STAGE:2022-12-01 10:55:55 81725:81725 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7735739Z STAGE:2022-12-01 10:55:55 81726:81726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7735847Z ok (7.743s)
2022-12-01T11:15:25.7735866Z 
2022-12-01T11:15:25.7736130Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7736246Z Ran 1 test in 7.744s
2022-12-01T11:15:25.7736265Z 
2022-12-01T11:15:25.7736357Z OK
2022-12-01T11:15:25.7736377Z 
2022-12-01T11:15:25.7736500Z Generating XML reports...
2022-12-01T11:15:25.7736954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105548.xml
2022-12-01T11:15:25.7737331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7737505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7737869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7738116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7738136Z 
2022-12-01T11:15:25.7738248Z Running tests...
2022-12-01T11:15:25.7738515Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7738831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7739073Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:15:25.7739092Z 
2022-12-01T11:15:25.7739360Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7739473Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7739493Z 
2022-12-01T11:15:25.7739582Z OK (skipped=1)
2022-12-01T11:15:25.7739617Z 
2022-12-01T11:15:25.7739721Z Generating XML reports...
2022-12-01T11:15:25.7740166Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105558.xml
2022-12-01T11:15:25.7740550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7740725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7741106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7741294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7741314Z 
2022-12-01T11:15:25.7741417Z Running tests...
2022-12-01T11:15:25.7741677Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7741973Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7742234Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:15:25.7742254Z 
2022-12-01T11:15:25.7742517Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7742681Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7742704Z 
2022-12-01T11:15:25.7742809Z OK (skipped=1)
2022-12-01T11:15:25.7742829Z 
2022-12-01T11:15:25.7742951Z Generating XML reports...
2022-12-01T11:15:25.7743404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105600.xml
2022-12-01T11:15:25.7743780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7743954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7744315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7744504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7744523Z 
2022-12-01T11:15:25.7744631Z Running tests...
2022-12-01T11:15:25.7744904Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7745223Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7745486Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T11:15:25.7745505Z 
2022-12-01T11:15:25.7745766Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7745880Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7745900Z 
2022-12-01T11:15:25.7746004Z OK (skipped=1)
2022-12-01T11:15:25.7746023Z 
2022-12-01T11:15:25.7746127Z Generating XML reports...
2022-12-01T11:15:25.7746573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105602.xml
2022-12-01T11:15:25.7746950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7747124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7747550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7747752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7747772Z 
2022-12-01T11:15:25.7747880Z Running tests...
2022-12-01T11:15:25.7748143Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7748438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7748713Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T11:15:25.7748732Z 
2022-12-01T11:15:25.7748990Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7749105Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7749124Z 
2022-12-01T11:15:25.7749229Z OK (skipped=1)
2022-12-01T11:15:25.7749251Z 
2022-12-01T11:15:25.7749548Z Generating XML reports...
2022-12-01T11:15:25.7750028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105605.xml
2022-12-01T11:15:25.7750403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7750580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7750942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7751132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7751153Z 
2022-12-01T11:15:25.7751266Z Running tests...
2022-12-01T11:15:25.7751526Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7751846Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7752187Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:15:25.7752210Z 
2022-12-01T11:15:25.7752474Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7752591Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7752610Z 
2022-12-01T11:15:25.7752715Z OK (skipped=1)
2022-12-01T11:15:25.7752734Z 
2022-12-01T11:15:25.7752838Z Generating XML reports...
2022-12-01T11:15:25.7753287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105607.xml
2022-12-01T11:15:25.7753663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7753844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7754223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7754416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7754435Z 
2022-12-01T11:15:25.7754545Z Running tests...
2022-12-01T11:15:25.7754806Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7755127Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7755383Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2022-12-01T11:15:25.7755419Z 
2022-12-01T11:15:25.7755668Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7755787Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7755806Z 
2022-12-01T11:15:25.7755912Z OK (skipped=1)
2022-12-01T11:15:25.7755931Z 
2022-12-01T11:15:25.7756053Z Generating XML reports...
2022-12-01T11:15:25.7756501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105609.xml
2022-12-01T11:15:25.7756937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7757120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7757507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7757681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7757700Z 
2022-12-01T11:15:25.7757808Z Running tests...
2022-12-01T11:15:25.7758075Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7758389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7758645Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2022-12-01T11:15:25.7758665Z 
2022-12-01T11:15:25.7758924Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7759039Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7759058Z 
2022-12-01T11:15:25.7759168Z OK (skipped=1)
2022-12-01T11:15:25.7759187Z 
2022-12-01T11:15:25.7759308Z Generating XML reports...
2022-12-01T11:15:25.7759742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105611.xml
2022-12-01T11:15:25.7760114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7760292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7760671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7760859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7760878Z 
2022-12-01T11:15:25.7760990Z Running tests...
2022-12-01T11:15:25.7761251Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7761633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7761891Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7761928Z 
2022-12-01T11:15:25.7762169Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7762283Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7762302Z 
2022-12-01T11:15:25.7762407Z OK (skipped=1)
2022-12-01T11:15:25.7762426Z 
2022-12-01T11:15:25.7762547Z Generating XML reports...
2022-12-01T11:15:25.7762992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105613.xml
2022-12-01T11:15:25.7763369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7763540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7763934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7764107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7764143Z 
2022-12-01T11:15:25.7764233Z Running tests...
2022-12-01T11:15:25.7764497Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7764817Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7765099Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7765119Z 
2022-12-01T11:15:25.7765382Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7765492Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7765511Z 
2022-12-01T11:15:25.7765617Z OK (skipped=1)
2022-12-01T11:15:25.7765636Z 
2022-12-01T11:15:25.7765766Z Generating XML reports...
2022-12-01T11:15:25.7766241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105615.xml
2022-12-01T11:15:25.7766625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7766804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7767181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7767371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7767390Z 
2022-12-01T11:15:25.7767497Z Running tests...
2022-12-01T11:15:25.7767761Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7768073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7768414Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7768438Z 
2022-12-01T11:15:25.7768681Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7768791Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7768811Z 
2022-12-01T11:15:25.7768915Z OK (skipped=1)
2022-12-01T11:15:25.7768933Z 
2022-12-01T11:15:25.7769056Z Generating XML reports...
2022-12-01T11:15:25.7769501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105617.xml
2022-12-01T11:15:25.7769871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7770045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7770426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7770671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7770691Z 
2022-12-01T11:15:25.7770783Z Running tests...
2022-12-01T11:15:25.7771049Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7771360Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7771651Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7771671Z 
2022-12-01T11:15:25.7771928Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7772037Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7772055Z 
2022-12-01T11:15:25.7772161Z OK (skipped=1)
2022-12-01T11:15:25.7772180Z 
2022-12-01T11:15:25.7772302Z Generating XML reports...
2022-12-01T11:15:25.7772733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105619.xml
2022-12-01T11:15:25.7773112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7773287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7773663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7773855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7773875Z 
2022-12-01T11:15:25.7773981Z Running tests...
2022-12-01T11:15:25.7774243Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7774552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7774853Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7774873Z 
2022-12-01T11:15:25.7775135Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7775277Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7775298Z 
2022-12-01T11:15:25.7775411Z OK (skipped=1)
2022-12-01T11:15:25.7775429Z 
2022-12-01T11:15:25.7775551Z Generating XML reports...
2022-12-01T11:15:25.7775995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105621.xml
2022-12-01T11:15:25.7776363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7776535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7776909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7777098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7777118Z 
2022-12-01T11:15:25.7777208Z Running tests...
2022-12-01T11:15:25.7777472Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7777786Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7778080Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.003s)
2022-12-01T11:15:25.7778100Z 
2022-12-01T11:15:25.7778360Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7778469Z Ran 1 test in 0.003s
2022-12-01T11:15:25.7778489Z 
2022-12-01T11:15:25.7778593Z OK (skipped=1)
2022-12-01T11:15:25.7778612Z 
2022-12-01T11:15:25.7778733Z Generating XML reports...
2022-12-01T11:15:25.7779177Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105623.xml
2022-12-01T11:15:25.7779530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7779773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7780164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7780359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7780378Z 
2022-12-01T11:15:25.7780485Z Running tests...
2022-12-01T11:15:25.7780748Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7781062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7781367Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7781386Z 
2022-12-01T11:15:25.7781651Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7781747Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7781770Z 
2022-12-01T11:15:25.7781876Z OK (skipped=1)
2022-12-01T11:15:25.7781894Z 
2022-12-01T11:15:25.7782018Z Generating XML reports...
2022-12-01T11:15:25.7782464Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105625.xml
2022-12-01T11:15:25.7782840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7783013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7783388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7783578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7783598Z 
2022-12-01T11:15:25.7783703Z Running tests...
2022-12-01T11:15:25.7783948Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7784261Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7784604Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7784625Z 
2022-12-01T11:15:25.7784894Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7785005Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7785024Z 
2022-12-01T11:15:25.7785130Z OK (skipped=1)
2022-12-01T11:15:25.7785149Z 
2022-12-01T11:15:25.7785271Z Generating XML reports...
2022-12-01T11:15:25.7785721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105627.xml
2022-12-01T11:15:25.7786074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7786246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7786623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7786826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7786845Z 
2022-12-01T11:15:25.7786953Z Running tests...
2022-12-01T11:15:25.7787212Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7787530Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7787830Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7787851Z 
2022-12-01T11:15:25.7788112Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7788210Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7788250Z 
2022-12-01T11:15:25.7788339Z OK (skipped=1)
2022-12-01T11:15:25.7788360Z 
2022-12-01T11:15:25.7788482Z Generating XML reports...
2022-12-01T11:15:25.7788931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105629.xml
2022-12-01T11:15:25.7789598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7789789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7790179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7790371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7790392Z 
2022-12-01T11:15:25.7790500Z Running tests...
2022-12-01T11:15:25.7790746Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7791056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7791341Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7791365Z 
2022-12-01T11:15:25.7791631Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7791742Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7791761Z 
2022-12-01T11:15:25.7791868Z OK (skipped=1)
2022-12-01T11:15:25.7791887Z 
2022-12-01T11:15:25.7792010Z Generating XML reports...
2022-12-01T11:15:25.7792450Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105631.xml
2022-12-01T11:15:25.7792821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7792981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7793358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7793548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7793571Z 
2022-12-01T11:15:25.7793678Z Running tests...
2022-12-01T11:15:25.7794010Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7794336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7794633Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7794652Z 
2022-12-01T11:15:25.7794910Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7795020Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7795039Z 
2022-12-01T11:15:25.7795130Z OK (skipped=1)
2022-12-01T11:15:25.7795148Z 
2022-12-01T11:15:25.7795270Z Generating XML reports...
2022-12-01T11:15:25.7795718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105633.xml
2022-12-01T11:15:25.7796088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7796271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7796657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7796848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7796868Z 
2022-12-01T11:15:25.7796977Z Running tests...
2022-12-01T11:15:25.7797225Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7797542Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7797840Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7797859Z 
2022-12-01T11:15:25.7798117Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7798297Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7798316Z 
2022-12-01T11:15:25.7798425Z OK (skipped=1)
2022-12-01T11:15:25.7798445Z 
2022-12-01T11:15:25.7798565Z Generating XML reports...
2022-12-01T11:15:25.7799012Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105635.xml
2022-12-01T11:15:25.7799379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7799535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7799919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7800107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7800127Z 
2022-12-01T11:15:25.7800234Z Running tests...
2022-12-01T11:15:25.7800494Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7800813Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7801120Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7801140Z 
2022-12-01T11:15:25.7801400Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7801510Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7801529Z 
2022-12-01T11:15:25.7801618Z OK (skipped=1)
2022-12-01T11:15:25.7801636Z 
2022-12-01T11:15:25.7801760Z Generating XML reports...
2022-12-01T11:15:25.7802200Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105637.xml
2022-12-01T11:15:25.7802570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7802744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7803173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7803372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7803391Z 
2022-12-01T11:15:25.7803498Z Running tests...
2022-12-01T11:15:25.7803761Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7804057Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7804355Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7804375Z 
2022-12-01T11:15:25.7804632Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7804742Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7804761Z 
2022-12-01T11:15:25.7804867Z OK (skipped=1)
2022-12-01T11:15:25.7804886Z 
2022-12-01T11:15:25.7805014Z Generating XML reports...
2022-12-01T11:15:25.7805465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105639.xml
2022-12-01T11:15:25.7805836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7806011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7806375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7806565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7806584Z 
2022-12-01T11:15:25.7806691Z Running tests...
2022-12-01T11:15:25.7806947Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7807260Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7807569Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7807636Z 
2022-12-01T11:15:25.7807905Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7808016Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7808035Z 
2022-12-01T11:15:25.7808123Z OK (skipped=1)
2022-12-01T11:15:25.7808159Z 
2022-12-01T11:15:25.7808263Z Generating XML reports...
2022-12-01T11:15:25.7808705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105642.xml
2022-12-01T11:15:25.7809077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7809252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7809631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7809825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7809847Z 
2022-12-01T11:15:25.7809953Z Running tests...
2022-12-01T11:15:25.7810218Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7810512Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7810806Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2022-12-01T11:15:25.7810826Z 
2022-12-01T11:15:25.7811083Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7811193Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7811212Z 
2022-12-01T11:15:25.7811318Z OK (skipped=1)
2022-12-01T11:15:25.7811337Z 
2022-12-01T11:15:25.7811458Z Generating XML reports...
2022-12-01T11:15:25.7811903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105644.xml
2022-12-01T11:15:25.7812333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7812521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7812891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7813080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7813100Z 
2022-12-01T11:15:25.7813206Z Running tests...
2022-12-01T11:15:25.7813468Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7813780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7814082Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2022-12-01T11:15:25.7814105Z 
2022-12-01T11:15:25.7814367Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7814479Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7814498Z 
2022-12-01T11:15:25.7814605Z OK (skipped=1)
2022-12-01T11:15:25.7814624Z 
2022-12-01T11:15:25.7814728Z Generating XML reports...
2022-12-01T11:15:25.7815171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105646.xml
2022-12-01T11:15:25.7815538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7815713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7816092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7816281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7816300Z 
2022-12-01T11:15:25.7816465Z Running tests...
2022-12-01T11:15:25.7816729Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7817026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7817291Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7817508Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84314
2022-12-01T11:15:25.7817727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84315
2022-12-01T11:15:25.7818097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7818268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7818647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7818836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7819209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7819365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7819740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7819928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7820175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7820419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7820819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7821217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7821497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7821733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7821957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7822198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7822602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7822998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7823101Z ok (6.941s)
2022-12-01T11:15:25.7823120Z 
2022-12-01T11:15:25.7823382Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7823498Z Ran 1 test in 6.941s
2022-12-01T11:15:25.7823517Z 
2022-12-01T11:15:25.7823611Z OK
2022-12-01T11:15:25.7823631Z 
2022-12-01T11:15:25.7823753Z Generating XML reports...
2022-12-01T11:15:25.7824189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105648.xml
2022-12-01T11:15:25.7824559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7824731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7825112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7825303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7825322Z 
2022-12-01T11:15:25.7825428Z Running tests...
2022-12-01T11:15:25.7825692Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7826073Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7826316Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7826535Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84747
2022-12-01T11:15:25.7826752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84748
2022-12-01T11:15:25.7827121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7827295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7827674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7827865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7828230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7828408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7828765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7828953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7829197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7829668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7830083Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7830477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7830712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7831012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7831154Z skip: Need at least 3 CUDA devices (3.913s)
2022-12-01T11:15:25.7831193Z 
2022-12-01T11:15:25.7831448Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7831559Z Ran 1 test in 3.913s
2022-12-01T11:15:25.7831578Z 
2022-12-01T11:15:25.7831682Z OK (skipped=1)
2022-12-01T11:15:25.7831702Z 
2022-12-01T11:15:25.7831823Z Generating XML reports...
2022-12-01T11:15:25.7832280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105657.xml
2022-12-01T11:15:25.7832650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7832823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7833206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7833379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7833415Z 
2022-12-01T11:15:25.7833506Z Running tests...
2022-12-01T11:15:25.7833770Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7834082Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7834333Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2022-12-01T11:15:25.7834353Z 
2022-12-01T11:15:25.7834614Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7834727Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7834747Z 
2022-12-01T11:15:25.7834854Z OK (skipped=1)
2022-12-01T11:15:25.7834872Z 
2022-12-01T11:15:25.7834994Z Generating XML reports...
2022-12-01T11:15:25.7835507Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105703.xml
2022-12-01T11:15:25.7835880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7836054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7836432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7836622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7836642Z 
2022-12-01T11:15:25.7836749Z Running tests...
2022-12-01T11:15:25.7837013Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7837331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7837553Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7837779Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85192
2022-12-01T11:15:25.7837996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85193
2022-12-01T11:15:25.7838369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7838543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7838920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7839109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7839473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7839646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7840053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7840251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7840498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7840739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7841142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7841542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7841771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7841998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7842086Z ok (4.741s)
2022-12-01T11:15:25.7842123Z 
2022-12-01T11:15:25.7842376Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7842489Z Ran 1 test in 4.741s
2022-12-01T11:15:25.7842508Z 
2022-12-01T11:15:25.7842601Z OK
2022-12-01T11:15:25.7842620Z 
2022-12-01T11:15:25.7842746Z Generating XML reports...
2022-12-01T11:15:25.7843199Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105705.xml
2022-12-01T11:15:25.7843574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7843749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7844133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7844306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7844386Z 
2022-12-01T11:15:25.7844501Z Running tests...
2022-12-01T11:15:25.7844774Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7845085Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7845335Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7845552Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85532
2022-12-01T11:15:25.7845767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85533
2022-12-01T11:15:25.7846137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7846295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7846676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7846867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7847237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7847409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7847784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7847967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7848211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7848454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7848841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7849319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7849561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7849789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7849890Z ok (6.408s)
2022-12-01T11:15:25.7849910Z 
2022-12-01T11:15:25.7850178Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7850289Z Ran 1 test in 6.408s
2022-12-01T11:15:25.7850308Z 
2022-12-01T11:15:25.7850399Z OK
2022-12-01T11:15:25.7850418Z 
2022-12-01T11:15:25.7850538Z Generating XML reports...
2022-12-01T11:15:25.7850972Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105712.xml
2022-12-01T11:15:25.7851345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7851523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7851905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7852094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7852113Z 
2022-12-01T11:15:25.7852223Z Running tests...
2022-12-01T11:15:25.7852489Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7852805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7853046Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7853265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85843
2022-12-01T11:15:25.7853482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85844
2022-12-01T11:15:25.7853853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7854084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7854468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7854659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7855024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7855197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7855554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7855745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7855987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7856238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7856640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7857036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7857264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7857492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7857712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7857953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7858353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7858795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7858906Z ok (4.833s)
2022-12-01T11:15:25.7858926Z 
2022-12-01T11:15:25.7859192Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7859304Z Ran 1 test in 4.833s
2022-12-01T11:15:25.7859324Z 
2022-12-01T11:15:25.7859416Z OK
2022-12-01T11:15:25.7859435Z 
2022-12-01T11:15:25.7859560Z Generating XML reports...
2022-12-01T11:15:25.7859995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105721.xml
2022-12-01T11:15:25.7860367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7860546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7860936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7861129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7861147Z 
2022-12-01T11:15:25.7861256Z Running tests...
2022-12-01T11:15:25.7861520Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7861834Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7862101Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7862303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86131
2022-12-01T11:15:25.7862519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86132
2022-12-01T11:15:25.7862891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7863123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7863507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7863697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7864061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7864234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7864596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7864784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7865030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7865274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7865681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7866077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7866307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7866534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7866692Z skip: Skipped due to small world size. (4.009s)
2022-12-01T11:15:25.7866713Z 
2022-12-01T11:15:25.7866961Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7867073Z Ran 1 test in 4.009s
2022-12-01T11:15:25.7867092Z 
2022-12-01T11:15:25.7867200Z OK (skipped=1)
2022-12-01T11:15:25.7867219Z 
2022-12-01T11:15:25.7867347Z Generating XML reports...
2022-12-01T11:15:25.7867842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105728.xml
2022-12-01T11:15:25.7868272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7868452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7868833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7869025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7869044Z 
2022-12-01T11:15:25.7869134Z Running tests...
2022-12-01T11:15:25.7869607Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7869940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7870190Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7870419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86426
2022-12-01T11:15:25.7870637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86427
2022-12-01T11:15:25.7871008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7871184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7871547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7871738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7872106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7872279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7872746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7872934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7873180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7873423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7873820Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7874200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7874430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7874658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7874820Z skip: Skipped due to small world size. (3.970s)
2022-12-01T11:15:25.7874843Z 
2022-12-01T11:15:25.7875110Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7875222Z Ran 1 test in 3.970s
2022-12-01T11:15:25.7875242Z 
2022-12-01T11:15:25.7875349Z OK (skipped=1)
2022-12-01T11:15:25.7875368Z 
2022-12-01T11:15:25.7875494Z Generating XML reports...
2022-12-01T11:15:25.7875927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105734.xml
2022-12-01T11:15:25.7876303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7876477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7876856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7877044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7877067Z 
2022-12-01T11:15:25.7877234Z Running tests...
2022-12-01T11:15:25.7877516Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7877831Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7878093Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7878293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86747
2022-12-01T11:15:25.7878509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86748
2022-12-01T11:15:25.7878881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7879055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7879433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7879632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7879998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7880170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7880527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7880715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7880961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7881204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7881602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7882064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7882294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7882521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7882680Z skip: Skipped due to small world size. (4.126s)
2022-12-01T11:15:25.7882700Z 
2022-12-01T11:15:25.7882949Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7883061Z Ran 1 test in 4.126s
2022-12-01T11:15:25.7883081Z 
2022-12-01T11:15:25.7883186Z OK (skipped=1)
2022-12-01T11:15:25.7883205Z 
2022-12-01T11:15:25.7883328Z Generating XML reports...
2022-12-01T11:15:25.7883776Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105740.xml
2022-12-01T11:15:25.7884156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7884333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7884711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7884899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7884919Z 
2022-12-01T11:15:25.7885007Z Running tests...
2022-12-01T11:15:25.7885272Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7885584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7885851Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7886070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86980
2022-12-01T11:15:25.7886289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86981
2022-12-01T11:15:25.7886704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7886888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7887251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7887442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7887801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7887976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7888354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7888538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7888791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7889032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7889433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7889813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7890042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7890269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7890508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7890748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7891205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7891598Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7891701Z ok (5.053s)
2022-12-01T11:15:25.7891720Z 
2022-12-01T11:15:25.7891986Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7892082Z Ran 1 test in 5.054s
2022-12-01T11:15:25.7892101Z 
2022-12-01T11:15:25.7892196Z OK
2022-12-01T11:15:25.7892215Z 
2022-12-01T11:15:25.7892340Z Generating XML reports...
2022-12-01T11:15:25.7892792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105747.xml
2022-12-01T11:15:25.7893163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7893344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7893727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7893917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7893937Z 
2022-12-01T11:15:25.7894046Z Running tests...
2022-12-01T11:15:25.7894293Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7894609Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7894876Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7895097Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87312
2022-12-01T11:15:25.7895313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87313
2022-12-01T11:15:25.7895734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7895918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7896302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7896475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7896838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7897009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7897382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7897571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7897821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7898069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7898471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7898868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7899080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7899307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7899546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7899785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7900243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7900638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7900740Z ok (5.106s)
2022-12-01T11:15:25.7900760Z 
2022-12-01T11:15:25.7901027Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7901138Z Ran 1 test in 5.106s
2022-12-01T11:15:25.7901158Z 
2022-12-01T11:15:25.7901233Z OK
2022-12-01T11:15:25.7901253Z 
2022-12-01T11:15:25.7901383Z Generating XML reports...
2022-12-01T11:15:25.7901837Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105754.xml
2022-12-01T11:15:25.7902213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7902399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7902803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7903004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7903024Z 
2022-12-01T11:15:25.7903132Z Running tests...
2022-12-01T11:15:25.7903393Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7903709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7903955Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7904173Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87672
2022-12-01T11:15:25.7904388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87673
2022-12-01T11:15:25.7904758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7904980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7905374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7905563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7905928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7906083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7906462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7906650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7906895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7907143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7907540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7907935Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7908164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7908391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7908533Z skip: Skipped due to small world size. (3.923s)
2022-12-01T11:15:25.7908552Z 
2022-12-01T11:15:25.7908816Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7908927Z Ran 1 test in 3.923s
2022-12-01T11:15:25.7908946Z 
2022-12-01T11:15:25.7909052Z OK (skipped=1)
2022-12-01T11:15:25.7909132Z 
2022-12-01T11:15:25.7909261Z Generating XML reports...
2022-12-01T11:15:25.7909958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105801.xml
2022-12-01T11:15:25.7910338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7910516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7910878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7911070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7911090Z 
2022-12-01T11:15:25.7911198Z Running tests...
2022-12-01T11:15:25.7911463Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7911777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7912050Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7912268Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87884
2022-12-01T11:15:25.7912485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87885
2022-12-01T11:15:25.7912858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7913015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7913391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7913580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7913943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7914117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7914563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7914763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7915007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7915231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7915634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7916033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7916263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7916490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7916597Z ok (3.971s)
2022-12-01T11:15:25.7916617Z 
2022-12-01T11:15:25.7916882Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7916993Z Ran 1 test in 3.971s
2022-12-01T11:15:25.7917013Z 
2022-12-01T11:15:25.7917105Z OK
2022-12-01T11:15:25.7917124Z 
2022-12-01T11:15:25.7917229Z Generating XML reports...
2022-12-01T11:15:25.7917678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105808.xml
2022-12-01T11:15:25.7918052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7918227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7918605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7918867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7918887Z 
2022-12-01T11:15:25.7918999Z Running tests...
2022-12-01T11:15:25.7919268Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7919587Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7919838Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7920057Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88196
2022-12-01T11:15:25.7920273Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88197
2022-12-01T11:15:25.7920646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7920822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7921202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7921399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7921767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7921924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7922300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7922544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7922796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7923040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7923439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7923887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7924127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7924354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7924437Z ok (4.055s)
2022-12-01T11:15:25.7924457Z 
2022-12-01T11:15:25.7924724Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7924841Z Ran 1 test in 4.056s
2022-12-01T11:15:25.7924860Z 
2022-12-01T11:15:25.7924952Z OK
2022-12-01T11:15:25.7924971Z 
2022-12-01T11:15:25.7925094Z Generating XML reports...
2022-12-01T11:15:25.7925546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105814.xml
2022-12-01T11:15:25.7925917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7926098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7926463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7926656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7926675Z 
2022-12-01T11:15:25.7926781Z Running tests...
2022-12-01T11:15:25.7927040Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7927352Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7927625Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:15:25.7927645Z 
2022-12-01T11:15:25.7927904Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7928070Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7928089Z 
2022-12-01T11:15:25.7928199Z OK (skipped=1)
2022-12-01T11:15:25.7928219Z 
2022-12-01T11:15:25.7928325Z Generating XML reports...
2022-12-01T11:15:25.7928779Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105820.xml
2022-12-01T11:15:25.7929149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7929325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7929709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7929897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7929916Z 
2022-12-01T11:15:25.7930022Z Running tests...
2022-12-01T11:15:25.7930285Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7930607Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7930847Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-12-01T11:15:25.7930868Z 
2022-12-01T11:15:25.7931131Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7931242Z Ran 1 test in 0.003s
2022-12-01T11:15:25.7931260Z 
2022-12-01T11:15:25.7931368Z OK (skipped=1)
2022-12-01T11:15:25.7931387Z 
2022-12-01T11:15:25.7931509Z Generating XML reports...
2022-12-01T11:15:25.7931957Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105822.xml
2022-12-01T11:15:25.7932331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7932504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7932935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7933115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7933134Z 
2022-12-01T11:15:25.7933244Z Running tests...
2022-12-01T11:15:25.7933511Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7933823Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7934093Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2022-12-01T11:15:25.7934112Z 
2022-12-01T11:15:25.7934371Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7934483Z Ran 1 test in 0.003s
2022-12-01T11:15:25.7934502Z 
2022-12-01T11:15:25.7934608Z OK (skipped=1)
2022-12-01T11:15:25.7934628Z 
2022-12-01T11:15:25.7934732Z Generating XML reports...
2022-12-01T11:15:25.7935192Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105824.xml
2022-12-01T11:15:25.7935569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7935745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7936126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7936319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7936339Z 
2022-12-01T11:15:25.7936445Z Running tests...
2022-12-01T11:15:25.7936709Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7937023Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7937264Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:15:25.7937350Z 
2022-12-01T11:15:25.7937604Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7937717Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7937736Z 
2022-12-01T11:15:25.7937843Z OK (skipped=1)
2022-12-01T11:15:25.7937861Z 
2022-12-01T11:15:25.7937987Z Generating XML reports...
2022-12-01T11:15:25.7938436Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105826.xml
2022-12-01T11:15:25.7938811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7938988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7939366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7939539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7939562Z 
2022-12-01T11:15:25.7939668Z Running tests...
2022-12-01T11:15:25.7939938Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7940255Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7940518Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:15:25.7940537Z 
2022-12-01T11:15:25.7940798Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7940907Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7940926Z 
2022-12-01T11:15:25.7941032Z OK (skipped=1)
2022-12-01T11:15:25.7941051Z 
2022-12-01T11:15:25.7941173Z Generating XML reports...
2022-12-01T11:15:25.7941600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105829.xml
2022-12-01T11:15:25.7941973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7942200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7942592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7942783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7942802Z 
2022-12-01T11:15:25.7942910Z Running tests...
2022-12-01T11:15:25.7943172Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7943483Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7943742Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:15:25.7943779Z 
2022-12-01T11:15:25.7944023Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7944139Z Ran 1 test in 0.003s
2022-12-01T11:15:25.7944159Z 
2022-12-01T11:15:25.7944264Z OK (skipped=1)
2022-12-01T11:15:25.7944286Z 
2022-12-01T11:15:25.7944408Z Generating XML reports...
2022-12-01T11:15:25.7944860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105831.xml
2022-12-01T11:15:25.7945232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7945405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7945781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7945954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7945991Z 
2022-12-01T11:15:25.7946081Z Running tests...
2022-12-01T11:15:25.7946349Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7946721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7946988Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.004s)
2022-12-01T11:15:25.7947008Z 
2022-12-01T11:15:25.7947272Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7947383Z Ran 1 test in 0.005s
2022-12-01T11:15:25.7947402Z 
2022-12-01T11:15:25.7947507Z OK (skipped=1)
2022-12-01T11:15:25.7947525Z 
2022-12-01T11:15:25.7947646Z Generating XML reports...
2022-12-01T11:15:25.7948078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105833.xml
2022-12-01T11:15:25.7948446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7948621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7949001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7949200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7949219Z 
2022-12-01T11:15:25.7949325Z Running tests...
2022-12-01T11:15:25.7949783Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7950100Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7950345Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2022-12-01T11:15:25.7950384Z 
2022-12-01T11:15:25.7950629Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7950740Z Ran 1 test in 0.002s
2022-12-01T11:15:25.7950759Z 
2022-12-01T11:15:25.7950865Z OK (skipped=1)
2022-12-01T11:15:25.7950883Z 
2022-12-01T11:15:25.7951005Z Generating XML reports...
2022-12-01T11:15:25.7951521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105835.xml
2022-12-01T11:15:25.7951910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7952085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7952462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7952634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7952671Z 
2022-12-01T11:15:25.7952759Z Running tests...
2022-12-01T11:15:25.7953018Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7953332Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7953577Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7953800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89102
2022-12-01T11:15:25.7954021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89103
2022-12-01T11:15:25.7954395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7954568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7954931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7955117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7955480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7955652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7956032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7956299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7956546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7956792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7957179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7960332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7960565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7960793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7961129Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7961464Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7962008Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7962028Z 
2022-12-01T11:15:25.7962356Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7962678Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7963015Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7963332Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7963659Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7964037Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7964381Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7964705Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7965030Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7965348Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7965681Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7966011Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7966324Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7966659Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7966994Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7967324Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7967650Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7967976Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7968355Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7968690Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7969072Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7969372Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7969704Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7970029Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7970354Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7970678Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7971005Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7971338Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7971670Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7971993Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7972305Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7972631Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7972957Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7973280Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7973612Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7973941Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7974312Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7974648Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7974978Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7975288Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7975608Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7975930Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7976256Z STAGE:2022-12-01 10:58:40 89103:89103 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7976582Z STAGE:2022-12-01 10:58:40 89102:89102 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7976690Z ok (3.839s)
2022-12-01T11:15:25.7976710Z 
2022-12-01T11:15:25.7976981Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7977096Z Ran 1 test in 3.840s
2022-12-01T11:15:25.7977115Z 
2022-12-01T11:15:25.7977188Z OK
2022-12-01T11:15:25.7977225Z 
2022-12-01T11:15:25.7977330Z Generating XML reports...
2022-12-01T11:15:25.7977784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105837.xml
2022-12-01T11:15:25.7978157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7978333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7978716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7978907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7978990Z 
2022-12-01T11:15:25.7979103Z Running tests...
2022-12-01T11:15:25.7979374Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7979674Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7979932Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7980687Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s)
2022-12-01T11:15:25.7980708Z 
2022-12-01T11:15:25.7980975Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7981087Z Ran 1 test in 1.657s
2022-12-01T11:15:25.7981106Z 
2022-12-01T11:15:25.7981212Z OK (skipped=1)
2022-12-01T11:15:25.7981235Z 
2022-12-01T11:15:25.7981358Z Generating XML reports...
2022-12-01T11:15:25.7981810Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105843.xml
2022-12-01T11:15:25.7982186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7982362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7982727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7982919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7982938Z 
2022-12-01T11:15:25.7983049Z Running tests...
2022-12-01T11:15:25.7983312Z ----------------------------------------------------------------------
2022-12-01T11:15:25.7983631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.7983946Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.7984174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89517
2022-12-01T11:15:25.7984393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89518
2022-12-01T11:15:25.7984749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7984925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7985307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7985499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7985863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.7986040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.7986416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.7986607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.7986853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.7987081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.7987484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7987881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.7988113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.7988401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.7988641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.7988883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.7989286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7989882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.7990209Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7990534Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7991079Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7991108Z 
2022-12-01T11:15:25.7991438Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7991760Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7992095Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7992422Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7992745Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7993074Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7993405Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7993791Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7994133Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7994455Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7994789Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7995117Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7995443Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7995762Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7996090Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7996413Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7996741Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7997059Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7997388Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7997713Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7998042Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7998363Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7998696Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7999158Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.7999463Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.7999789Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8000118Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8000445Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8000767Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8001084Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8001415Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8001749Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8002078Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8002381Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8002707Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8003037Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8003361Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8003682Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8004011Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8004388Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8004724Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8005043Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8005355Z STAGE:2022-12-01 10:58:50 89518:89518 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8005677Z STAGE:2022-12-01 10:58:50 89517:89517 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8005779Z ok (3.909s)
2022-12-01T11:15:25.8005799Z 
2022-12-01T11:15:25.8006065Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8006179Z Ran 1 test in 3.909s
2022-12-01T11:15:25.8006198Z 
2022-12-01T11:15:25.8006290Z OK
2022-12-01T11:15:25.8006313Z 
2022-12-01T11:15:25.8006436Z Generating XML reports...
2022-12-01T11:15:25.8006894Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105847.xml
2022-12-01T11:15:25.8007271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8007429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8007812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8008006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8008026Z 
2022-12-01T11:15:25.8008135Z Running tests...
2022-12-01T11:15:25.8008399Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8008714Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8009025Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8009247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89813
2022-12-01T11:15:25.8009449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89814
2022-12-01T11:15:25.8009825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8009997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8010376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8010565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8010931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8011106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8011490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8011679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8011906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8012147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8012550Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8012948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8013177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8013402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8013609Z skip: Skipped due to small world size. (4.035s)
2022-12-01T11:15:25.8013630Z 
2022-12-01T11:15:25.8013909Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8014004Z Ran 1 test in 4.035s
2022-12-01T11:15:25.8014042Z 
2022-12-01T11:15:25.8014131Z OK (skipped=1)
2022-12-01T11:15:25.8014150Z 
2022-12-01T11:15:25.8014274Z Generating XML reports...
2022-12-01T11:15:25.8014728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105853.xml
2022-12-01T11:15:25.8015101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8015277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8015657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8015858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8015880Z 
2022-12-01T11:15:25.8015991Z Running tests...
2022-12-01T11:15:25.8016238Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8016551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8016809Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8017028Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90127
2022-12-01T11:15:25.8017245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90128
2022-12-01T11:15:25.8017615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8017794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8018176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8018406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8018775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8018949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8019331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8019517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8019765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8020010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8020411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8020816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8021031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8021259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8022033Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:15:25.8022149Z   warnings.warn(
2022-12-01T11:15:25.8022956Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1342: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2022-12-01T11:15:25.8023077Z   warnings.warn(
2022-12-01T11:15:25.8023178Z ok (5.785s)
2022-12-01T11:15:25.8023198Z 
2022-12-01T11:15:25.8023466Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8023578Z Ran 1 test in 5.785s
2022-12-01T11:15:25.8023597Z 
2022-12-01T11:15:25.8023690Z OK
2022-12-01T11:15:25.8023709Z 
2022-12-01T11:15:25.8023814Z Generating XML reports...
2022-12-01T11:15:25.8024268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105859.xml
2022-12-01T11:15:25.8024644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8024822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8025201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8025399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8025418Z 
2022-12-01T11:15:25.8025525Z Running tests...
2022-12-01T11:15:25.8025791Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8026090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8026355Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8027105Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.623s)
2022-12-01T11:15:25.8027126Z 
2022-12-01T11:15:25.8027390Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8027553Z Ran 1 test in 1.623s
2022-12-01T11:15:25.8027575Z 
2022-12-01T11:15:25.8027683Z OK (skipped=1)
2022-12-01T11:15:25.8027702Z 
2022-12-01T11:15:25.8027824Z Generating XML reports...
2022-12-01T11:15:25.8028278Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105907.xml
2022-12-01T11:15:25.8028650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8028825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8029187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8029610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8029632Z 
2022-12-01T11:15:25.8029748Z Running tests...
2022-12-01T11:15:25.8030027Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8030346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8030659Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8031404Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.699s)
2022-12-01T11:15:25.8031424Z 
2022-12-01T11:15:25.8031687Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8031799Z Ran 1 test in 1.700s
2022-12-01T11:15:25.8031818Z 
2022-12-01T11:15:25.8031923Z OK (skipped=1)
2022-12-01T11:15:25.8031942Z 
2022-12-01T11:15:25.8032048Z Generating XML reports...
2022-12-01T11:15:25.8032580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105911.xml
2022-12-01T11:15:25.8032972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8033147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8033527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8033718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8033738Z 
2022-12-01T11:15:25.8033844Z Running tests...
2022-12-01T11:15:25.8034108Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8034402Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8034721Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8035478Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.742s)
2022-12-01T11:15:25.8035498Z 
2022-12-01T11:15:25.8035760Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8035872Z Ran 1 test in 1.742s
2022-12-01T11:15:25.8035892Z 
2022-12-01T11:15:25.8035999Z OK (skipped=1)
2022-12-01T11:15:25.8036018Z 
2022-12-01T11:15:25.8036143Z Generating XML reports...
2022-12-01T11:15:25.8036592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105915.xml
2022-12-01T11:15:25.8036964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8037212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8037581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8037772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8037792Z 
2022-12-01T11:15:25.8037900Z Running tests...
2022-12-01T11:15:25.8038166Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8038481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8038752Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8038973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90846
2022-12-01T11:15:25.8039192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90847
2022-12-01T11:15:25.8039578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8039736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8040115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8040307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8040670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8040842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8041217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8041406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8041657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8041932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8042347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8042744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8042976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8043203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8043460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfwejm8oq
2022-12-01T11:15:25.8043731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfwejm8oq/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8043992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0pwe9wu
2022-12-01T11:15:25.8044262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0pwe9wu/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8044480Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8044716Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8044819Z ok (6.317s)
2022-12-01T11:15:25.8044839Z 
2022-12-01T11:15:25.8045110Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8045223Z Ran 1 test in 6.318s
2022-12-01T11:15:25.8045243Z 
2022-12-01T11:15:25.8045335Z OK
2022-12-01T11:15:25.8045353Z 
2022-12-01T11:15:25.8045476Z Generating XML reports...
2022-12-01T11:15:25.8045928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105919.xml
2022-12-01T11:15:25.8046363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8046542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8046923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8047115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8047134Z 
2022-12-01T11:15:25.8047241Z Running tests...
2022-12-01T11:15:25.8047505Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8047818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8048094Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8048315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91184
2022-12-01T11:15:25.8048520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91185
2022-12-01T11:15:25.8048892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8049068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8049446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8049633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8049996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8050167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8050539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8050713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8051005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8051260Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8051663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8052062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8052288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8052514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8052769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfipv0d2v
2022-12-01T11:15:25.8053046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfipv0d2v/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8053282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4fqv8v80
2022-12-01T11:15:25.8053550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4fqv8v80/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8053786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8054015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8054249Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8054479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8054579Z ok (6.197s)
2022-12-01T11:15:25.8054599Z 
2022-12-01T11:15:25.8054869Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8054963Z Ran 1 test in 6.197s
2022-12-01T11:15:25.8055046Z 
2022-12-01T11:15:25.8055121Z OK
2022-12-01T11:15:25.8055140Z 
2022-12-01T11:15:25.8055268Z Generating XML reports...
2022-12-01T11:15:25.8055728Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105928.xml
2022-12-01T11:15:25.8056099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8056273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8056652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8056841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8056861Z 
2022-12-01T11:15:25.8056967Z Running tests...
2022-12-01T11:15:25.8057215Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8057533Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8057814Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8058561Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.676s)
2022-12-01T11:15:25.8058582Z 
2022-12-01T11:15:25.8058847Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8058958Z Ran 1 test in 1.676s
2022-12-01T11:15:25.8058977Z 
2022-12-01T11:15:25.8059084Z OK (skipped=1)
2022-12-01T11:15:25.8059103Z 
2022-12-01T11:15:25.8059225Z Generating XML reports...
2022-12-01T11:15:25.8059671Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105936.xml
2022-12-01T11:15:25.8060102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8060267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8060650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8060842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8060861Z 
2022-12-01T11:15:25.8060969Z Running tests...
2022-12-01T11:15:25.8061234Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8061545Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8061833Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8062575Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2022-12-01T11:15:25.8062599Z 
2022-12-01T11:15:25.8062864Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8062957Z Ran 1 test in 1.641s
2022-12-01T11:15:25.8062995Z 
2022-12-01T11:15:25.8063086Z OK (skipped=1)
2022-12-01T11:15:25.8063104Z 
2022-12-01T11:15:25.8063227Z Generating XML reports...
2022-12-01T11:15:25.8063677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105940.xml
2022-12-01T11:15:25.8064050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8064224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8064667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8064856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8064876Z 
2022-12-01T11:15:25.8064982Z Running tests...
2022-12-01T11:15:25.8065227Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8065540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8065825Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8066045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91832
2022-12-01T11:15:25.8066261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91833
2022-12-01T11:15:25.8066632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8066816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8067196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8067386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8067738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8067913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8068286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8068525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8068775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8069019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8069714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8070149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8070361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8070590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8070846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpftefeeea
2022-12-01T11:15:25.8071117Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpftefeeea/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8071369Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9d0ge2r
2022-12-01T11:15:25.8071640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9d0ge2r/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8071856Z <class 'torch.nn.parameter.Parameter'>
2022-12-01T11:15:25.8071957Z ok (5.504s)
2022-12-01T11:15:25.8071978Z 
2022-12-01T11:15:25.8072245Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8072340Z Ran 1 test in 5.504s
2022-12-01T11:15:25.8072359Z 
2022-12-01T11:15:25.8072451Z OK
2022-12-01T11:15:25.8072470Z 
2022-12-01T11:15:25.8072594Z Generating XML reports...
2022-12-01T11:15:25.8073045Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105944.xml
2022-12-01T11:15:25.8073420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8073597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8073980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8074247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8074266Z 
2022-12-01T11:15:25.8074356Z Running tests...
2022-12-01T11:15:25.8074621Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8074933Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8075241Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8075460Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92120
2022-12-01T11:15:25.8075675Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92121
2022-12-01T11:15:25.8076050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8076231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8076614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8076789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8077156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8077329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8077703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8077896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8078141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8078386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8078837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8079245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8079458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8079686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8079948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpae_p_v
2022-12-01T11:15:25.8080219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpae_p_v/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8080473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptr2votq_
2022-12-01T11:15:25.8080738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptr2votq_/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8080843Z ok (5.644s)
2022-12-01T11:15:25.8080865Z 
2022-12-01T11:15:25.8081136Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8081230Z Ran 1 test in 5.644s
2022-12-01T11:15:25.8081249Z 
2022-12-01T11:15:25.8081341Z OK
2022-12-01T11:15:25.8081360Z 
2022-12-01T11:15:25.8081484Z Generating XML reports...
2022-12-01T11:15:25.8081933Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105952.xml
2022-12-01T11:15:25.8082306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8082480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8082863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8083055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8083120Z 
2022-12-01T11:15:25.8083234Z Running tests...
2022-12-01T11:15:25.8083483Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8083797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8084057Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8084275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92426
2022-12-01T11:15:25.8084490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92427
2022-12-01T11:15:25.8084863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8085040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8085427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8085606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8085978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8086149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8086528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8086718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8086963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8087204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8087603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8088050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8088269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8088497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8088750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7qslmba
2022-12-01T11:15:25.8089019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7qslmba/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8089269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_jlf487
2022-12-01T11:15:25.8089533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_jlf487/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8089767Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8090009Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8090093Z ok (6.201s)
2022-12-01T11:15:25.8090130Z 
2022-12-01T11:15:25.8090387Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8090500Z Ran 1 test in 6.201s
2022-12-01T11:15:25.8090519Z 
2022-12-01T11:15:25.8090610Z OK
2022-12-01T11:15:25.8090629Z 
2022-12-01T11:15:25.8090753Z Generating XML reports...
2022-12-01T11:15:25.8091206Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110000.xml
2022-12-01T11:15:25.8091580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8091757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8092141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8092373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8092392Z 
2022-12-01T11:15:25.8092501Z Running tests...
2022-12-01T11:15:25.8092766Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8093078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8093366Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8093584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92766
2022-12-01T11:15:25.8093802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92767
2022-12-01T11:15:25.8094172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8094330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8094717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8094908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8095276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8095449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8095826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8096015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8096261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8096502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8096933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8097341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8097571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8097801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8098056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3_tiwvl
2022-12-01T11:15:25.8098320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3_tiwvl/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8098570Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3v35d2qv
2022-12-01T11:15:25.8098835Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3v35d2qv/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8099621Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.8099727Z ok (6.126s)
2022-12-01T11:15:25.8099747Z 
2022-12-01T11:15:25.8100019Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8100114Z Ran 1 test in 6.126s
2022-12-01T11:15:25.8100133Z 
2022-12-01T11:15:25.8100225Z OK
2022-12-01T11:15:25.8100244Z 
2022-12-01T11:15:25.8100367Z Generating XML reports...
2022-12-01T11:15:25.8100822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110008.xml
2022-12-01T11:15:25.8101255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8101431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8101819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8102010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8102030Z 
2022-12-01T11:15:25.8102120Z Running tests...
2022-12-01T11:15:25.8102385Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8102699Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8102979Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8103729Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.652s)
2022-12-01T11:15:25.8103750Z 
2022-12-01T11:15:25.8104013Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8104124Z Ran 1 test in 1.652s
2022-12-01T11:15:25.8104143Z 
2022-12-01T11:15:25.8104253Z OK (skipped=1)
2022-12-01T11:15:25.8104272Z 
2022-12-01T11:15:25.8104394Z Generating XML reports...
2022-12-01T11:15:25.8104845Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110017.xml
2022-12-01T11:15:25.8105200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8105378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8105803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8106000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8106019Z 
2022-12-01T11:15:25.8106126Z Running tests...
2022-12-01T11:15:25.8106396Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8106709Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8106966Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8107169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93170
2022-12-01T11:15:25.8107390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93171
2022-12-01T11:15:25.8107763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8107945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8108316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8108487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8108868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8109060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8109671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8109852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8110099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8110435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8110850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8111246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8111476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8111704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8111958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13v_7bl1
2022-12-01T11:15:25.8112214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5zguyq8
2022-12-01T11:15:25.8112461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13v_7bl1/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8112729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5zguyq8/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8113624Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8114511Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8115721Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T11:15:25.8115966Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T11:15:25.8117132Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1064.)
2022-12-01T11:15:25.8117368Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2022-12-01T11:15:25.8117606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8117839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8118722Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8119607Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8120535Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8121400Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8122282Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8123151Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8124058Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8124938Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8125803Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8126671Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2022-12-01T11:15:25.8126776Z ok (3.889s)
2022-12-01T11:15:25.8126797Z 
2022-12-01T11:15:25.8127064Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8127178Z Ran 1 test in 3.889s
2022-12-01T11:15:25.8127198Z 
2022-12-01T11:15:25.8127289Z OK
2022-12-01T11:15:25.8127308Z 
2022-12-01T11:15:25.8127430Z Generating XML reports...
2022-12-01T11:15:25.8127868Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110020.xml
2022-12-01T11:15:25.8128245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8128478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8128864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8129056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8129075Z 
2022-12-01T11:15:25.8129183Z Running tests...
2022-12-01T11:15:25.8129448Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8129763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8130011Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8130736Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.671s)
2022-12-01T11:15:25.8130780Z 
2022-12-01T11:15:25.8131031Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8131144Z Ran 1 test in 1.671s
2022-12-01T11:15:25.8131164Z 
2022-12-01T11:15:25.8131269Z OK (skipped=1)
2022-12-01T11:15:25.8131288Z 
2022-12-01T11:15:25.8131411Z Generating XML reports...
2022-12-01T11:15:25.8131863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110027.xml
2022-12-01T11:15:25.8132239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8132410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8132791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8132983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8133006Z 
2022-12-01T11:15:25.8133096Z Running tests...
2022-12-01T11:15:25.8133405Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8133731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8133997Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8134216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93580
2022-12-01T11:15:25.8134431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93581
2022-12-01T11:15:25.8134801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8134974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8135337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8135533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8135898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8136075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8136447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8136639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8136885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8137129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8137531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8137974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8138204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8138435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8138691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6x9r68v
2022-12-01T11:15:25.8138959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6x9r68v/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8139209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpabgwbo2c
2022-12-01T11:15:25.8139475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpabgwbo2c/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8140272Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T11:15:25.8140610Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T11:15:25.8141394Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1113: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2022-12-01T11:15:25.8141724Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2022-12-01T11:15:25.8141809Z ok (6.189s)
2022-12-01T11:15:25.8141829Z 
2022-12-01T11:15:25.8142095Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8142211Z Ran 1 test in 6.190s
2022-12-01T11:15:25.8142230Z 
2022-12-01T11:15:25.8142403Z OK
2022-12-01T11:15:25.8142424Z 
2022-12-01T11:15:25.8142554Z Generating XML reports...
2022-12-01T11:15:25.8143006Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110030.xml
2022-12-01T11:15:25.8143377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8143553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8143914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8144105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8144125Z 
2022-12-01T11:15:25.8144233Z Running tests...
2022-12-01T11:15:25.8144496Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8144816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8145084Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8145828Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.715s)
2022-12-01T11:15:25.8145848Z 
2022-12-01T11:15:25.8146115Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8146226Z Ran 1 test in 1.716s
2022-12-01T11:15:25.8146245Z 
2022-12-01T11:15:25.8146349Z OK (skipped=1)
2022-12-01T11:15:25.8146368Z 
2022-12-01T11:15:25.8146474Z Generating XML reports...
2022-12-01T11:15:25.8146926Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110039.xml
2022-12-01T11:15:25.8147358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8147538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8147918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8148107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8148126Z 
2022-12-01T11:15:25.8148233Z Running tests...
2022-12-01T11:15:25.8148496Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8148793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8149063Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8150006Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.682s)
2022-12-01T11:15:25.8150030Z 
2022-12-01T11:15:25.8150305Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8150418Z Ran 1 test in 1.682s
2022-12-01T11:15:25.8150438Z 
2022-12-01T11:15:25.8150546Z OK (skipped=1)
2022-12-01T11:15:25.8150564Z 
2022-12-01T11:15:25.8150688Z Generating XML reports...
2022-12-01T11:15:25.8151137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110043.xml
2022-12-01T11:15:25.8151510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8151685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8152122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8152323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8152342Z 
2022-12-01T11:15:25.8152449Z Running tests...
2022-12-01T11:15:25.8152711Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8153024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8153313Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8153534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94284
2022-12-01T11:15:25.8153753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94285
2022-12-01T11:15:25.8154124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8154290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8154674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8154865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8155230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8155403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8155776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8155966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8156216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8156514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8156921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8157319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8157548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8157776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8158017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8158262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8158659Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8159060Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8159298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp51cvovtx
2022-12-01T11:15:25.8159568Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp51cvovtx/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8159824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk6r8jdyb
2022-12-01T11:15:25.8160148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk6r8jdyb/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8160382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8160615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8160848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8161125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8161234Z ok (6.704s)
2022-12-01T11:15:25.8161254Z 
2022-12-01T11:15:25.8161507Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8161621Z Ran 1 test in 6.705s
2022-12-01T11:15:25.8161640Z 
2022-12-01T11:15:25.8161732Z OK
2022-12-01T11:15:25.8161751Z 
2022-12-01T11:15:25.8161875Z Generating XML reports...
2022-12-01T11:15:25.8162330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110047.xml
2022-12-01T11:15:25.8162703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8162876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8163258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8163438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8163475Z 
2022-12-01T11:15:25.8163565Z Running tests...
2022-12-01T11:15:25.8163825Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8164142Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8164418Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8164638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94576
2022-12-01T11:15:25.8164857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94577
2022-12-01T11:15:25.8165226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8165402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8165826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8166019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8166384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8166554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8166935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8167124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8167369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8167613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8168003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8168448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8168680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8168909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8169187Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:15:25.8169462Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:15:25.8169716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hy5uo8q
2022-12-01T11:15:25.8169984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hy5uo8q/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8170291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp615zd1k6
2022-12-01T11:15:25.8170547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp615zd1k6/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8170784Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8171016Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8171243Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8171469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8171745Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:15:25.8172019Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:15:25.8172299Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:15:25.8172567Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2022-12-01T11:15:25.8172782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8173015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8173244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8173472Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8173743Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:15:25.8174020Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2022-12-01T11:15:25.8174369Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T11:15:25.8174635Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2022-12-01T11:15:25.8174864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8175078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8175309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8175543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8175644Z ok (6.840s)
2022-12-01T11:15:25.8175663Z 
2022-12-01T11:15:25.8175936Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8176048Z Ran 1 test in 6.841s
2022-12-01T11:15:25.8176071Z 
2022-12-01T11:15:25.8176166Z OK
2022-12-01T11:15:25.8176184Z 
2022-12-01T11:15:25.8176309Z Generating XML reports...
2022-12-01T11:15:25.8176746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110056.xml
2022-12-01T11:15:25.8177116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8177290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8177672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8177863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8177882Z 
2022-12-01T11:15:25.8177991Z Running tests...
2022-12-01T11:15:25.8178260Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8178575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8178905Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8179654Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2022-12-01T11:15:25.8179692Z 
2022-12-01T11:15:25.8179941Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8180056Z Ran 1 test in 1.630s
2022-12-01T11:15:25.8180076Z 
2022-12-01T11:15:25.8180183Z OK (skipped=1)
2022-12-01T11:15:25.8180202Z 
2022-12-01T11:15:25.8180327Z Generating XML reports...
2022-12-01T11:15:25.8180778Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110105.xml
2022-12-01T11:15:25.8181160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8181335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8181721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8181914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8181933Z 
2022-12-01T11:15:25.8182023Z Running tests...
2022-12-01T11:15:25.8182290Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8182604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8182876Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8183093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95087
2022-12-01T11:15:25.8183371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95088
2022-12-01T11:15:25.8183743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8183917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8184279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8184469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8184833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8185004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8185382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8185577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8185826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8186073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8186472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8186867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8187097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8187309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8187901Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T11:15:25.8188452Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2022-12-01T11:15:25.8188707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1feiy09h
2022-12-01T11:15:25.8188976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1feiy09h/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8189231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2dmno19
2022-12-01T11:15:25.8189729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2dmno19/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8189980Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8190218Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8190492Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T11:15:25.8190762Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2022-12-01T11:15:25.8191043Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T11:15:25.8191338Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2022-12-01T11:15:25.8191666Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T11:15:25.8192078Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2022-12-01T11:15:25.8192401Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T11:15:25.8192716Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2022-12-01T11:15:25.8192950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8193184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8193285Z ok (6.016s)
2022-12-01T11:15:25.8193306Z 
2022-12-01T11:15:25.8193588Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8193688Z Ran 1 test in 6.016s
2022-12-01T11:15:25.8193707Z 
2022-12-01T11:15:25.8193801Z OK
2022-12-01T11:15:25.8193823Z 
2022-12-01T11:15:25.8193947Z Generating XML reports...
2022-12-01T11:15:25.8194399Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110109.xml
2022-12-01T11:15:25.8194773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8194951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8195331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8195522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8195542Z 
2022-12-01T11:15:25.8195631Z Running tests...
2022-12-01T11:15:25.8195894Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8196209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8196583Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8196817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95443
2022-12-01T11:15:25.8197030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95444
2022-12-01T11:15:25.8197403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8197578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8197959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8198133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8198505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8198679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8199054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8199239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8199487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8199734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8200133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8200515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8200801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8201031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8201289Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzrfrty3r
2022-12-01T11:15:25.8201562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzrfrty3r/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8201816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnl1x62mq
2022-12-01T11:15:25.8202082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnl1x62mq/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8202320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8202553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8202768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8202996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8203094Z ok (6.387s)
2022-12-01T11:15:25.8203114Z 
2022-12-01T11:15:25.8203387Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8203494Z Ran 1 test in 6.387s
2022-12-01T11:15:25.8203513Z 
2022-12-01T11:15:25.8203593Z OK
2022-12-01T11:15:25.8203612Z 
2022-12-01T11:15:25.8203732Z Generating XML reports...
2022-12-01T11:15:25.8204183Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110117.xml
2022-12-01T11:15:25.8204551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8204709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8205090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8205327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8205348Z 
2022-12-01T11:15:25.8205458Z Running tests...
2022-12-01T11:15:25.8205724Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8206036Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8206344Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8206564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95778
2022-12-01T11:15:25.8206764Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95779
2022-12-01T11:15:25.8207136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8207317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8207695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8207884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8208244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8208415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8208790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8208977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8209207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8209450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8209911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8210305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8210531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8210848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8211101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoa2kw3hq
2022-12-01T11:15:25.8211367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoa2kw3hq/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8211601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvq5br5j9
2022-12-01T11:15:25.8211868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvq5br5j9/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8212109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8212344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8212575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8212805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8212907Z ok (6.527s)
2022-12-01T11:15:25.8212927Z 
2022-12-01T11:15:25.8213193Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8213309Z Ran 1 test in 6.528s
2022-12-01T11:15:25.8213328Z 
2022-12-01T11:15:25.8213401Z OK
2022-12-01T11:15:25.8213420Z 
2022-12-01T11:15:25.8213539Z Generating XML reports...
2022-12-01T11:15:25.8213989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110125.xml
2022-12-01T11:15:25.8214410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8214592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8214972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8215162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8215182Z 
2022-12-01T11:15:25.8215286Z Running tests...
2022-12-01T11:15:25.8215532Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8215848Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8216217Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8216444Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96180
2022-12-01T11:15:25.8216661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96181
2022-12-01T11:15:25.8217032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8217206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8217585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8217775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8218120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8218293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8218668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8218914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8219157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8219401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8219800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8220190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8220423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8220634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8220892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7g8y0218
2022-12-01T11:15:25.8221158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7g8y0218/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8221410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvpms_nmf
2022-12-01T11:15:25.8221676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvpms_nmf/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8221910Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8222147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8222439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8222677Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8222762Z ok (16.853s)
2022-12-01T11:15:25.8222782Z 
2022-12-01T11:15:25.8223056Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8223217Z Ran 1 test in 16.854s
2022-12-01T11:15:25.8223238Z 
2022-12-01T11:15:25.8223329Z OK
2022-12-01T11:15:25.8223348Z 
2022-12-01T11:15:25.8223473Z Generating XML reports...
2022-12-01T11:15:25.8223925Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110134.xml
2022-12-01T11:15:25.8224298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8224475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8224839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8225032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8225052Z 
2022-12-01T11:15:25.8225160Z Running tests...
2022-12-01T11:15:25.8225427Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8225742Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8226110Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8226328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96743
2022-12-01T11:15:25.8226548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96744
2022-12-01T11:15:25.8226918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8227079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8227457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8227702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8228073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8228244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8228611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8228798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8229043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8229268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8229903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8230311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8230542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8230770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8231026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswmv6xct
2022-12-01T11:15:25.8231291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswmv6xct/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8231544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbdws8qzk
2022-12-01T11:15:25.8231811Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbdws8qzk/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8232027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8232267Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8232575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8232820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8232919Z ok (6.486s)
2022-12-01T11:15:25.8232939Z 
2022-12-01T11:15:25.8233209Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8233320Z Ran 1 test in 6.486s
2022-12-01T11:15:25.8233340Z 
2022-12-01T11:15:25.8233431Z OK
2022-12-01T11:15:25.8233449Z 
2022-12-01T11:15:25.8233554Z Generating XML reports...
2022-12-01T11:15:25.8234007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110153.xml
2022-12-01T11:15:25.8234378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8234553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8234935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8235126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8235145Z 
2022-12-01T11:15:25.8235250Z Running tests...
2022-12-01T11:15:25.8235512Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8235828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8236179Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8236400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97079
2022-12-01T11:15:25.8236612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97080
2022-12-01T11:15:25.8237072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8237245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8237628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8237816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8238179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8238354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8238713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8238901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8239148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8239396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8239793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8240190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8240420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8240644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8240900Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1iim6in
2022-12-01T11:15:25.8241151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1iim6in/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8241456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1nl1gk4
2022-12-01T11:15:25.8241726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1nl1gk4/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8241958Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8242195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8242426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8242654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8242756Z ok (6.431s)
2022-12-01T11:15:25.8242776Z 
2022-12-01T11:15:25.8243029Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8243141Z Ran 1 test in 6.432s
2022-12-01T11:15:25.8243160Z 
2022-12-01T11:15:25.8243251Z OK
2022-12-01T11:15:25.8243274Z 
2022-12-01T11:15:25.8243394Z Generating XML reports...
2022-12-01T11:15:25.8243852Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110202.xml
2022-12-01T11:15:25.8244223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8244400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8244778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8244969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8244988Z 
2022-12-01T11:15:25.8245078Z Running tests...
2022-12-01T11:15:25.8245344Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8245659Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8246086Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8246303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97450
2022-12-01T11:15:25.8246522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97451
2022-12-01T11:15:25.8246895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8247069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8247432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8247619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8247983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8248166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8248542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8248731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8248973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8249218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8249620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8250001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8250229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8250503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8250769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4i63y4rt
2022-12-01T11:15:25.8251038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4i63y4rt/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8251291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmm4wu2y
2022-12-01T11:15:25.8251559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmm4wu2y/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8251795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8252030Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8252246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8252479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8252581Z ok (6.397s)
2022-12-01T11:15:25.8252600Z 
2022-12-01T11:15:25.8252870Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8252983Z Ran 1 test in 6.397s
2022-12-01T11:15:25.8253003Z 
2022-12-01T11:15:25.8253094Z OK
2022-12-01T11:15:25.8253113Z 
2022-12-01T11:15:25.8253234Z Generating XML reports...
2022-12-01T11:15:25.8253686Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110211.xml
2022-12-01T11:15:25.8254040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8254217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8254602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8254845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8254864Z 
2022-12-01T11:15:25.8254972Z Running tests...
2022-12-01T11:15:25.8255239Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8255553Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8255923Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8256141Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97844
2022-12-01T11:15:25.8256340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97845
2022-12-01T11:15:25.8256713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8256889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8257272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8257461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8257826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8257999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8258372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8258543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8258788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8259034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8259481Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8259886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8260117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8260342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8260600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52lur7i1
2022-12-01T11:15:25.8260865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52lur7i1/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8261103Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyyo5l75t
2022-12-01T11:15:25.8261370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyyo5l75t/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8261613Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8261849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8262080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8262305Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8262405Z ok (6.453s)
2022-12-01T11:15:25.8262425Z 
2022-12-01T11:15:25.8262693Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8262802Z Ran 1 test in 6.453s
2022-12-01T11:15:25.8262822Z 
2022-12-01T11:15:25.8262895Z OK
2022-12-01T11:15:25.8262914Z 
2022-12-01T11:15:25.8263035Z Generating XML reports...
2022-12-01T11:15:25.8263487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110219.xml
2022-12-01T11:15:25.8263914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8264088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8264465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8264658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8264677Z 
2022-12-01T11:15:25.8264783Z Running tests...
2022-12-01T11:15:25.8265026Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8265344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8265712Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8265936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98187
2022-12-01T11:15:25.8266151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98188
2022-12-01T11:15:25.8266523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8266696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8267080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8267274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8267623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8267791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8268161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8268443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8268702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8268945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8269508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8269928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8270157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8270370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8270626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpak8hivzq
2022-12-01T11:15:25.8270903Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpak8hivzq/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8271151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjr1vnqy
2022-12-01T11:15:25.8271416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjr1vnqy/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8271650Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8271884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8272117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8272326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8272427Z ok (6.345s)
2022-12-01T11:15:25.8272447Z 
2022-12-01T11:15:25.8272713Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8272912Z Ran 1 test in 6.345s
2022-12-01T11:15:25.8272934Z 
2022-12-01T11:15:25.8273025Z OK
2022-12-01T11:15:25.8273044Z 
2022-12-01T11:15:25.8273168Z Generating XML reports...
2022-12-01T11:15:25.8273619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110228.xml
2022-12-01T11:15:25.8273990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8274164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8274524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8274715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8274734Z 
2022-12-01T11:15:25.8274839Z Running tests...
2022-12-01T11:15:25.8275099Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8275420Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8275789Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8276011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98510
2022-12-01T11:15:25.8276229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98511
2022-12-01T11:15:25.8276601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8276758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8277134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8277324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8277754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8277934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8278310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8278498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8278740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8278965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8279363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8279761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8279996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8280223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8280482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd65u983_
2022-12-01T11:15:25.8280747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd65u983_/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8281004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1b2l56qp
2022-12-01T11:15:25.8281274Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1b2l56qp/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8281491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8281725Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8282055Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8282280Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8282382Z ok (6.345s)
2022-12-01T11:15:25.8282402Z 
2022-12-01T11:15:25.8282671Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8282783Z Ran 1 test in 6.345s
2022-12-01T11:15:25.8282802Z 
2022-12-01T11:15:25.8282894Z OK
2022-12-01T11:15:25.8282912Z 
2022-12-01T11:15:25.8283016Z Generating XML reports...
2022-12-01T11:15:25.8283470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110237.xml
2022-12-01T11:15:25.8283840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8284014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8284403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8284595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8284614Z 
2022-12-01T11:15:25.8284719Z Running tests...
2022-12-01T11:15:25.8284981Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8285294Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8285642Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8285863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98890
2022-12-01T11:15:25.8286079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98891
2022-12-01T11:15:25.8286449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8286675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8287063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8287253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8287619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8287791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8288149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8288337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8288581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8288829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8289228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8289624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8289853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8290082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8290322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa846wvqj
2022-12-01T11:15:25.8290591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa846wvqj/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8290845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx7buh600
2022-12-01T11:15:25.8291166Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx7buh600/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8291401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8291636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8291868Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8292095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8292192Z ok (6.422s)
2022-12-01T11:15:25.8292211Z 
2022-12-01T11:15:25.8292464Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8292578Z Ran 1 test in 6.423s
2022-12-01T11:15:25.8292597Z 
2022-12-01T11:15:25.8292690Z OK
2022-12-01T11:15:25.8292709Z 
2022-12-01T11:15:25.8292831Z Generating XML reports...
2022-12-01T11:15:25.8293289Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110245.xml
2022-12-01T11:15:25.8293661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8293835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8294216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8294388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8294424Z 
2022-12-01T11:15:25.8294514Z Running tests...
2022-12-01T11:15:25.8294776Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8295090Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8295397Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8295672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99265
2022-12-01T11:15:25.8295895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99266
2022-12-01T11:15:25.8296262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8296438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8296798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8296989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8297345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8297516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8297899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8298091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8298336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8298578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8298982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8299359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8299587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8299814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8300142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpelinwj2p
2022-12-01T11:15:25.8300412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpelinwj2p/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8300667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvub7cpbg
2022-12-01T11:15:25.8300934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvub7cpbg/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8301169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8301384Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8301615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8301845Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8301951Z ok (6.435s)
2022-12-01T11:15:25.8301971Z 
2022-12-01T11:15:25.8302246Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8302358Z Ran 1 test in 6.435s
2022-12-01T11:15:25.8302377Z 
2022-12-01T11:15:25.8302468Z OK
2022-12-01T11:15:25.8302486Z 
2022-12-01T11:15:25.8302608Z Generating XML reports...
2022-12-01T11:15:25.8303058Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110254.xml
2022-12-01T11:15:25.8303412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8303595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8303974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8304164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8304188Z 
2022-12-01T11:15:25.8304294Z Running tests...
2022-12-01T11:15:25.8304598Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8304921Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8305231Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8305432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99644
2022-12-01T11:15:25.8305651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99645
2022-12-01T11:15:25.8306020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8306193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8306571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8306768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8307135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8307307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8307683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8307856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8308100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8308345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8308742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8309192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8309654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8309892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8310149Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk06jg_h5
2022-12-01T11:15:25.8310418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk06jg_h5/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8310653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwf58pu17
2022-12-01T11:15:25.8310920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwf58pu17/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8311155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8311394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8311626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8311856Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8311952Z ok (6.412s)
2022-12-01T11:15:25.8311973Z 
2022-12-01T11:15:25.8312247Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8312342Z Ran 1 test in 6.412s
2022-12-01T11:15:25.8312361Z 
2022-12-01T11:15:25.8312453Z OK
2022-12-01T11:15:25.8312472Z 
2022-12-01T11:15:25.8312593Z Generating XML reports...
2022-12-01T11:15:25.8313048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110302.xml
2022-12-01T11:15:25.8313415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8313590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8314039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8314241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8314260Z 
2022-12-01T11:15:25.8314366Z Running tests...
2022-12-01T11:15:25.8314615Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8314925Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8315188Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8315937Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.703s)
2022-12-01T11:15:25.8315961Z 
2022-12-01T11:15:25.8316227Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8316338Z Ran 1 test in 1.703s
2022-12-01T11:15:25.8316357Z 
2022-12-01T11:15:25.8316464Z OK (skipped=1)
2022-12-01T11:15:25.8316483Z 
2022-12-01T11:15:25.8316604Z Generating XML reports...
2022-12-01T11:15:25.8317051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110311.xml
2022-12-01T11:15:25.8317403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8317579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8317959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8318150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8318236Z 
2022-12-01T11:15:25.8318355Z Running tests...
2022-12-01T11:15:25.8318626Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8318940Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8319193Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8319414Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100235
2022-12-01T11:15:25.8319616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100236
2022-12-01T11:15:25.8319989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8320164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8320542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8320737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8321099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8321271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8321644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8321814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8322059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8322303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8322700Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8323144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8323379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8323601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8323859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprbezn89l
2022-12-01T11:15:25.8324127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprbezn89l/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8324360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_j6og3ej
2022-12-01T11:15:25.8324625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_j6og3ej/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8324726Z ok (6.522s)
2022-12-01T11:15:25.8324745Z 
2022-12-01T11:15:25.8325018Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8325140Z Ran 1 test in 6.523s
2022-12-01T11:15:25.8325159Z 
2022-12-01T11:15:25.8325252Z OK
2022-12-01T11:15:25.8325271Z 
2022-12-01T11:15:25.8325390Z Generating XML reports...
2022-12-01T11:15:25.8325841Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110315.xml
2022-12-01T11:15:25.8326212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8326369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8326750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8326941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8326960Z 
2022-12-01T11:15:25.8327066Z Running tests...
2022-12-01T11:15:25.8327330Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8327703Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8327975Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8328196Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100594
2022-12-01T11:15:25.8328396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100602
2022-12-01T11:15:25.8328763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8328940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8329314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8329505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8329878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8330049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8330423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8330611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8330840Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8331081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8331478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8331876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8332154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8332388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8332642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncenga1r
2022-12-01T11:15:25.8332907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncenga1r/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8333142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7aq27e4
2022-12-01T11:15:25.8333407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7aq27e4/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8333638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8333871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8334285Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpv7aq27e4'>
2022-12-01T11:15:25.8334450Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:15:25.8334843Z /opt/conda/lib/python3.10/tempfile.py:837: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpncenga1r'>
2022-12-01T11:15:25.8335003Z   _warnings.warn(warn_message, ResourceWarning)
2022-12-01T11:15:25.8335104Z ok (6.080s)
2022-12-01T11:15:25.8335124Z 
2022-12-01T11:15:25.8335369Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8335483Z Ran 1 test in 6.080s
2022-12-01T11:15:25.8335502Z 
2022-12-01T11:15:25.8335591Z OK
2022-12-01T11:15:25.8335610Z 
2022-12-01T11:15:25.8335733Z Generating XML reports...
2022-12-01T11:15:25.8336185Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110324.xml
2022-12-01T11:15:25.8336556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8336792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8337170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8337344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8337382Z 
2022-12-01T11:15:25.8337472Z Running tests...
2022-12-01T11:15:25.8337733Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8338048Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8338312Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8338531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100871
2022-12-01T11:15:25.8338746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100873
2022-12-01T11:15:25.8339119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8339295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8339661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8339851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8340213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8340385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8340757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8340945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8341241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8341492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8341877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8342270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8342500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8342754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4on5u_f3
2022-12-01T11:15:25.8343020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4on5u_f3/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8343245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8343504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp59m538i7
2022-12-01T11:15:25.8343767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp59m538i7/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8344000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8344216Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8344319Z ok (4.181s)
2022-12-01T11:15:25.8344339Z 
2022-12-01T11:15:25.8344608Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8344721Z Ran 1 test in 4.181s
2022-12-01T11:15:25.8344740Z 
2022-12-01T11:15:25.8344831Z OK
2022-12-01T11:15:25.8344849Z 
2022-12-01T11:15:25.8344972Z Generating XML reports...
2022-12-01T11:15:25.8345423Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110332.xml
2022-12-01T11:15:25.8345853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8346029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8346390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8346580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8346599Z 
2022-12-01T11:15:25.8346707Z Running tests...
2022-12-01T11:15:25.8346971Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8347281Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8347546Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8347767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101261
2022-12-01T11:15:25.8347993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101262
2022-12-01T11:15:25.8348346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8348522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8348900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8349087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8349673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8349856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8350240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8350433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8350753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8350988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8351391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8351789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8352018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8352241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8352497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiciafhsi
2022-12-01T11:15:25.8352770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiciafhsi/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8353023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps74d1zl1
2022-12-01T11:15:25.8353272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps74d1zl1/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8353509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8353738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8353839Z ok (6.119s)
2022-12-01T11:15:25.8353859Z 
2022-12-01T11:15:25.8354127Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8354240Z Ran 1 test in 6.119s
2022-12-01T11:15:25.8354259Z 
2022-12-01T11:15:25.8354349Z OK
2022-12-01T11:15:25.8354368Z 
2022-12-01T11:15:25.8354492Z Generating XML reports...
2022-12-01T11:15:25.8354949Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110339.xml
2022-12-01T11:15:25.8355403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8355578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8355956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8356143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8356162Z 
2022-12-01T11:15:25.8356268Z Running tests...
2022-12-01T11:15:25.8356529Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8356842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8357131Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8357342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101515
2022-12-01T11:15:25.8357562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101516
2022-12-01T11:15:25.8357935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8358109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8358488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8358675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8359040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8359209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8359585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8359807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8360060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8360303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8360703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8361097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8361325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8361549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8361789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8362034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8362413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8362805Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8363041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.8363280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.8363663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8364053Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8364374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0awn576z
2022-12-01T11:15:25.8364645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0awn576z/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8364897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47mbj_a1
2022-12-01T11:15:25.8365144Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47mbj_a1/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8365246Z ok (5.595s)
2022-12-01T11:15:25.8365266Z 
2022-12-01T11:15:25.8365534Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8365647Z Ran 1 test in 5.595s
2022-12-01T11:15:25.8365666Z 
2022-12-01T11:15:25.8365757Z OK
2022-12-01T11:15:25.8365776Z 
2022-12-01T11:15:25.8365897Z Generating XML reports...
2022-12-01T11:15:25.8366352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110347.xml
2022-12-01T11:15:25.8366723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8366899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8367266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8367454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8367474Z 
2022-12-01T11:15:25.8367579Z Running tests...
2022-12-01T11:15:25.8367838Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8368148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8368477Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8368704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101856
2022-12-01T11:15:25.8368970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101857
2022-12-01T11:15:25.8369336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8369512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8369891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8370082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8370446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8370618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8370990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8371187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8371433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8371661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8372060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8372457Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8372686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8372916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8373155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8373450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8373845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8374233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8374453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.8374687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.8375071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8375465Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8375728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcouqvb46
2022-12-01T11:15:25.8375998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcouqvb46/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8376250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpd6e7ng
2022-12-01T11:15:25.8376516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpd6e7ng/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8376615Z ok (15.746s)
2022-12-01T11:15:25.8376635Z 
2022-12-01T11:15:25.8376885Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8376998Z Ran 1 test in 15.746s
2022-12-01T11:15:25.8377018Z 
2022-12-01T11:15:25.8377108Z OK
2022-12-01T11:15:25.8377127Z 
2022-12-01T11:15:25.8377250Z Generating XML reports...
2022-12-01T11:15:25.8377703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110355.xml
2022-12-01T11:15:25.8378121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8378301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8378682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8378854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8378890Z 
2022-12-01T11:15:25.8378981Z Running tests...
2022-12-01T11:15:25.8379241Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8379552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8379862Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8380087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102473
2022-12-01T11:15:25.8380303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102474
2022-12-01T11:15:25.8380672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8380848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8381210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8381401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8381765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8381937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8382307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8382553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8382800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8383042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8383425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8383821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8384049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8384277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8384533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv0toix1
2022-12-01T11:15:25.8384810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv0toix1/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8385061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp52vef00r
2022-12-01T11:15:25.8385322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp52vef00r/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8385425Z ok (6.494s)
2022-12-01T11:15:25.8385444Z 
2022-12-01T11:15:25.8385695Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8385805Z Ran 1 test in 6.494s
2022-12-01T11:15:25.8385825Z 
2022-12-01T11:15:25.8385917Z OK
2022-12-01T11:15:25.8385936Z 
2022-12-01T11:15:25.8386057Z Generating XML reports...
2022-12-01T11:15:25.8386508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110413.xml
2022-12-01T11:15:25.8386879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8387104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8387492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8387681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8387701Z 
2022-12-01T11:15:25.8387791Z Running tests...
2022-12-01T11:15:25.8388050Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8388361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8388648Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8388869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102729
2022-12-01T11:15:25.8389093Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102730
2022-12-01T11:15:25.8389691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8389874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8390242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8390437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8390798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8390972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8391345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8391533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8391865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8392110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8392507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8392887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8393114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8393339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8393596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpge1tol6j
2022-12-01T11:15:25.8393863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpge1tol6j/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8394122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvef86o8
2022-12-01T11:15:25.8394389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvef86o8/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8394493Z ok (6.474s)
2022-12-01T11:15:25.8394513Z 
2022-12-01T11:15:25.8394783Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8394879Z Ran 1 test in 6.474s
2022-12-01T11:15:25.8394898Z 
2022-12-01T11:15:25.8394990Z OK
2022-12-01T11:15:25.8395009Z 
2022-12-01T11:15:25.8395131Z Generating XML reports...
2022-12-01T11:15:25.8395583Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110422.xml
2022-12-01T11:15:25.8395952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8396127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8396569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8396768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8396787Z 
2022-12-01T11:15:25.8396878Z Running tests...
2022-12-01T11:15:25.8397144Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8397456Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8397712Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8397932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103151
2022-12-01T11:15:25.8398151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103152
2022-12-01T11:15:25.8398525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8398708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8399068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8399260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8399625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8399800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8400183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8400372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8400613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8400916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8401316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8423031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8423315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8423541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8423796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1h7nmgi
2022-12-01T11:15:25.8424055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1h7nmgi/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8424300Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4k7uc2p4
2022-12-01T11:15:25.8424569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4k7uc2p4/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8424661Z ok (6.014s)
2022-12-01T11:15:25.8424683Z 
2022-12-01T11:15:25.8424984Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8425086Z Ran 1 test in 6.014s
2022-12-01T11:15:25.8425106Z 
2022-12-01T11:15:25.8425179Z OK
2022-12-01T11:15:25.8425198Z 
2022-12-01T11:15:25.8425312Z Generating XML reports...
2022-12-01T11:15:25.8425765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110430.xml
2022-12-01T11:15:25.8426132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8426299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8426672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8426958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8426980Z 
2022-12-01T11:15:25.8427086Z Running tests...
2022-12-01T11:15:25.8427343Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8427652Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8427903Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8428112Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103514
2022-12-01T11:15:25.8428320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103515
2022-12-01T11:15:25.8428683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8428848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8429229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8429706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8430088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8430253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8430624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8430801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8431035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8431266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8431778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8432167Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8432383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8432595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8432840Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsc242i33
2022-12-01T11:15:25.8433095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsc242i33/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8433335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyuzkg09u
2022-12-01T11:15:25.8433588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyuzkg09u/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8434360Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.8435129Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.8435422Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8435659Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8435750Z ok (6.146s)
2022-12-01T11:15:25.8435771Z 
2022-12-01T11:15:25.8436035Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8436137Z Ran 1 test in 6.147s
2022-12-01T11:15:25.8436157Z 
2022-12-01T11:15:25.8436231Z OK
2022-12-01T11:15:25.8436250Z 
2022-12-01T11:15:25.8436372Z Generating XML reports...
2022-12-01T11:15:25.8436818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110439.xml
2022-12-01T11:15:25.8437181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8437349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8437721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8437901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8437922Z 
2022-12-01T11:15:25.8438019Z Running tests...
2022-12-01T11:15:25.8438272Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8438577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8438852Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8439601Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2022-12-01T11:15:25.8439678Z 
2022-12-01T11:15:25.8439939Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8440039Z Ran 1 test in 1.645s
2022-12-01T11:15:25.8440059Z 
2022-12-01T11:15:25.8440162Z OK (skipped=1)
2022-12-01T11:15:25.8440181Z 
2022-12-01T11:15:25.8440299Z Generating XML reports...
2022-12-01T11:15:25.8440749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110447.xml
2022-12-01T11:15:25.8441123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8441282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8441657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8441836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8441860Z 
2022-12-01T11:15:25.8441960Z Running tests...
2022-12-01T11:15:25.8442213Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8442523Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8442803Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8443549Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.748s)
2022-12-01T11:15:25.8443570Z 
2022-12-01T11:15:25.8443832Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8443943Z Ran 1 test in 1.748s
2022-12-01T11:15:25.8443967Z 
2022-12-01T11:15:25.8444055Z OK (skipped=1)
2022-12-01T11:15:25.8444074Z 
2022-12-01T11:15:25.8444278Z Generating XML reports...
2022-12-01T11:15:25.8444742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110451.xml
2022-12-01T11:15:25.8445115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8445288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8445670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8445859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8445879Z 
2022-12-01T11:15:25.8445986Z Running tests...
2022-12-01T11:15:25.8446230Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8446547Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8446830Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8447048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104084
2022-12-01T11:15:25.8447267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104085
2022-12-01T11:15:25.8447637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8447811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8448190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8448380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8448730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8448979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8449364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8449553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8449798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8450041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8450443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8450843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8451070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8451289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8451544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1k3lqsde
2022-12-01T11:15:25.8451812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1k3lqsde/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8452063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplx2ay55c
2022-12-01T11:15:25.8452328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplx2ay55c/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8452666Z STAGE:2022-12-01 11:05:01 104084:104084 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8452989Z STAGE:2022-12-01 11:05:01 104085:104085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8453228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8453496Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8454061Z STAGE:2022-12-01 11:05:01 104085:104085 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:05:01 104084:104084 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8454081Z 
2022-12-01T11:15:25.8454855Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.8455652Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.8456196Z STAGE:2022-12-01 11:05:01 104084:104084 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 11:05:01 104085:104085 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8456217Z 
2022-12-01T11:15:25.8456765Z STAGE:2022-12-01 11:05:01 104085:104085 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:05:01 104084:104084 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8456786Z 
2022-12-01T11:15:25.8456919Z ok (6.692s)
2022-12-01T11:15:25.8456954Z 
2022-12-01T11:15:25.8457208Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8457320Z Ran 1 test in 6.692s
2022-12-01T11:15:25.8457339Z 
2022-12-01T11:15:25.8457430Z OK
2022-12-01T11:15:25.8457449Z 
2022-12-01T11:15:25.8457572Z Generating XML reports...
2022-12-01T11:15:25.8458025Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110455.xml
2022-12-01T11:15:25.8458397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8458572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8458951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8459128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8459151Z 
2022-12-01T11:15:25.8459259Z Running tests...
2022-12-01T11:15:25.8459529Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8459843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8460110Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8460331Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104427
2022-12-01T11:15:25.8460549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104428
2022-12-01T11:15:25.8460919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8461076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8461459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8461653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8462064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8462245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8462623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8462811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8463055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8463298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8463687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8464091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8464321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8464548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8464804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspjnfpay
2022-12-01T11:15:25.8465079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspjnfpay/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8465332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprleyercd
2022-12-01T11:15:25.8465597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprleyercd/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8465699Z ok (5.542s)
2022-12-01T11:15:25.8465719Z 
2022-12-01T11:15:25.8465972Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8466138Z Ran 1 test in 5.542s
2022-12-01T11:15:25.8466158Z 
2022-12-01T11:15:25.8466249Z OK
2022-12-01T11:15:25.8466269Z 
2022-12-01T11:15:25.8466392Z Generating XML reports...
2022-12-01T11:15:25.8466844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110504.xml
2022-12-01T11:15:25.8467220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8467393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8467772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8467944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8467980Z 
2022-12-01T11:15:25.8468071Z Running tests...
2022-12-01T11:15:25.8468333Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8468705Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8468980Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8469914Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.672s)
2022-12-01T11:15:25.8469937Z 
2022-12-01T11:15:25.8470210Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8470322Z Ran 1 test in 1.672s
2022-12-01T11:15:25.8470343Z 
2022-12-01T11:15:25.8470449Z OK (skipped=1)
2022-12-01T11:15:25.8470468Z 
2022-12-01T11:15:25.8470591Z Generating XML reports...
2022-12-01T11:15:25.8471028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110512.xml
2022-12-01T11:15:25.8471484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8471671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8472054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8472244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8472264Z 
2022-12-01T11:15:25.8472370Z Running tests...
2022-12-01T11:15:25.8472633Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8472947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8473209Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8473435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104840
2022-12-01T11:15:25.8473657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104842
2022-12-01T11:15:25.8474028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8474201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8474578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8474767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8475136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8475309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8475673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8475935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8476181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8476425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8476827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8477224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8477453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8477678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8477934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjwmx1bi
2022-12-01T11:15:25.8478195Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjwmx1bi/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8478449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt61peuvo
2022-12-01T11:15:25.8478716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt61peuvo/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8479629Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T11:15:25.8479741Z   warnings.warn(
2022-12-01T11:15:25.8480690Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1772: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2022-12-01T11:15:25.8480810Z   warnings.warn(
2022-12-01T11:15:25.8481049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8481281Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8481382Z ok (6.208s)
2022-12-01T11:15:25.8481402Z 
2022-12-01T11:15:25.8481656Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8481767Z Ran 1 test in 6.208s
2022-12-01T11:15:25.8481786Z 
2022-12-01T11:15:25.8481876Z OK
2022-12-01T11:15:25.8481895Z 
2022-12-01T11:15:25.8482017Z Generating XML reports...
2022-12-01T11:15:25.8482465Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110516.xml
2022-12-01T11:15:25.8482844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8483020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8483402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8483592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8483612Z 
2022-12-01T11:15:25.8483702Z Running tests...
2022-12-01T11:15:25.8483968Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8484283Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8484558Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8485364Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2022-12-01T11:15:25.8485385Z 
2022-12-01T11:15:25.8485646Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8485757Z Ran 1 test in 1.665s
2022-12-01T11:15:25.8485776Z 
2022-12-01T11:15:25.8485880Z OK (skipped=1)
2022-12-01T11:15:25.8485899Z 
2022-12-01T11:15:25.8486020Z Generating XML reports...
2022-12-01T11:15:25.8486449Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110524.xml
2022-12-01T11:15:25.8486821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8486994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8487382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8487572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8487592Z 
2022-12-01T11:15:25.8487697Z Running tests...
2022-12-01T11:15:25.8487956Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8488267Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8488537Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8488742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105405
2022-12-01T11:15:25.8488962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105406
2022-12-01T11:15:25.8489334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8489557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8489946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8490137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8490502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8490674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8491036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8491227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8491470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8491724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8492126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8492524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8492754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8492983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8493241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp965y97uy
2022-12-01T11:15:25.8493491Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp965y97uy/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8493745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvn7cvv8
2022-12-01T11:15:25.8494073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvn7cvv8/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8494417Z STAGE:2022-12-01 11:05:33 105406:105406 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8494740Z STAGE:2022-12-01 11:05:33 105405:105405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8494977Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8495209Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2022-12-01T11:15:25.8495763Z STAGE:2022-12-01 11:05:34 105406:105406 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:05:34 105405:105405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8495783Z 
2022-12-01T11:15:25.8496118Z STAGE:2022-12-01 11:05:34 105405:105405 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8496465Z STAGE:2022-12-01 11:05:34 105405:105405 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8496552Z ok (6.875s)
2022-12-01T11:15:25.8496571Z 
2022-12-01T11:15:25.8496836Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8496948Z Ran 1 test in 6.875s
2022-12-01T11:15:25.8496968Z 
2022-12-01T11:15:25.8497059Z OK
2022-12-01T11:15:25.8497078Z 
2022-12-01T11:15:25.8497201Z Generating XML reports...
2022-12-01T11:15:25.8497653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110528.xml
2022-12-01T11:15:25.8498025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8498202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8498568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8498808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8498829Z 
2022-12-01T11:15:25.8498942Z Running tests...
2022-12-01T11:15:25.8499208Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8499519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8499779Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8499999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105870
2022-12-01T11:15:25.8500216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105871
2022-12-01T11:15:25.8500589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8500748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8501139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8501330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8501695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8501866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8502241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8502427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8502669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8502894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8503356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8503754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8503984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8504207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8504462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspdld70g
2022-12-01T11:15:25.8504730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspdld70g/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8504980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkjbsibd
2022-12-01T11:15:25.8505251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkjbsibd/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8505340Z ok (5.637s)
2022-12-01T11:15:25.8505360Z 
2022-12-01T11:15:25.8505634Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8505746Z Ran 1 test in 5.637s
2022-12-01T11:15:25.8505765Z 
2022-12-01T11:15:25.8505857Z OK
2022-12-01T11:15:25.8505876Z 
2022-12-01T11:15:25.8505997Z Generating XML reports...
2022-12-01T11:15:25.8506448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110537.xml
2022-12-01T11:15:25.8506821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8506997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8507362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8507553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8507576Z 
2022-12-01T11:15:25.8507684Z Running tests...
2022-12-01T11:15:25.8507990Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8508312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8508580Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8508798Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106181
2022-12-01T11:15:25.8509014Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106182
2022-12-01T11:15:25.8509623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8509797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8510181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8510380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8510747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8510922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8511301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8511492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8511738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8511964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8512365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8512873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8513102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8513326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8513582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdtjqc5dl
2022-12-01T11:15:25.8513850Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdtjqc5dl/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8514101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmzbqsll9
2022-12-01T11:15:25.8514366Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmzbqsll9/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8514451Z ok (5.440s)
2022-12-01T11:15:25.8514489Z 
2022-12-01T11:15:25.8514741Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8514857Z Ran 1 test in 5.440s
2022-12-01T11:15:25.8514877Z 
2022-12-01T11:15:25.8514971Z OK
2022-12-01T11:15:25.8514990Z 
2022-12-01T11:15:25.8515113Z Generating XML reports...
2022-12-01T11:15:25.8515563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110545.xml
2022-12-01T11:15:25.8515933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8516106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8516486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8516660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8516680Z 
2022-12-01T11:15:25.8516786Z Running tests...
2022-12-01T11:15:25.8517048Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8517425Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8517708Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8518456Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.689s)
2022-12-01T11:15:25.8518477Z 
2022-12-01T11:15:25.8518743Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8518858Z Ran 1 test in 1.689s
2022-12-01T11:15:25.8518878Z 
2022-12-01T11:15:25.8518981Z OK (skipped=1)
2022-12-01T11:15:25.8519000Z 
2022-12-01T11:15:25.8519105Z Generating XML reports...
2022-12-01T11:15:25.8519557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110553.xml
2022-12-01T11:15:25.8519939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8520115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8520496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8520685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8520704Z 
2022-12-01T11:15:25.8520811Z Running tests...
2022-12-01T11:15:25.8521072Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8521388Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8521629Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8522492Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.799s)
2022-12-01T11:15:25.8522513Z 
2022-12-01T11:15:25.8522781Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8522892Z Ran 1 test in 1.799s
2022-12-01T11:15:25.8522912Z 
2022-12-01T11:15:25.8523019Z OK (skipped=1)
2022-12-01T11:15:25.8523038Z 
2022-12-01T11:15:25.8523159Z Generating XML reports...
2022-12-01T11:15:25.8523611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110557.xml
2022-12-01T11:15:25.8523986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8524160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8524544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8524719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8524738Z 
2022-12-01T11:15:25.8524846Z Running tests...
2022-12-01T11:15:25.8525107Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8525419Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8525707Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8526443Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.669s)
2022-12-01T11:15:25.8526467Z 
2022-12-01T11:15:25.8526775Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8526893Z Ran 1 test in 1.669s
2022-12-01T11:15:25.8526913Z 
2022-12-01T11:15:25.8527017Z OK (skipped=1)
2022-12-01T11:15:25.8527036Z 
2022-12-01T11:15:25.8527142Z Generating XML reports...
2022-12-01T11:15:25.8527596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110601.xml
2022-12-01T11:15:25.8527965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8528140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8528516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8528705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8528729Z 
2022-12-01T11:15:25.8528834Z Running tests...
2022-12-01T11:15:25.8529098Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8529415Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8529692Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8529912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106842
2022-12-01T11:15:25.8530130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106843
2022-12-01T11:15:25.8530502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8530675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8531055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8531303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8531672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8531844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8532205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8532395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8532637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8532879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8533279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8533685Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8533912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8534138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8534376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpz8ymjd
2022-12-01T11:15:25.8534645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpz8ymjd/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8534894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe61fm6i4
2022-12-01T11:15:25.8535154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe61fm6i4/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8535255Z ok (12.217s)
2022-12-01T11:15:25.8535276Z 
2022-12-01T11:15:25.8535545Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8535704Z Ran 1 test in 12.217s
2022-12-01T11:15:25.8535725Z 
2022-12-01T11:15:25.8535819Z OK
2022-12-01T11:15:25.8535838Z 
2022-12-01T11:15:25.8535963Z Generating XML reports...
2022-12-01T11:15:25.8536400Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110605.xml
2022-12-01T11:15:25.8536776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8536950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8537331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8537519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8537539Z 
2022-12-01T11:15:25.8537646Z Running tests...
2022-12-01T11:15:25.8537910Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8538231Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8538473Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8538694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107249
2022-12-01T11:15:25.8538911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107250
2022-12-01T11:15:25.8539279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8539451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8539826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8540013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8540442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8540617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8540978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8541166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8541410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8541650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8542050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8542446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8542683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8542908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8543152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8543372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8543767Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8544161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8544261Z ok (4.017s)
2022-12-01T11:15:25.8544281Z 
2022-12-01T11:15:25.8544544Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8544658Z Ran 1 test in 4.017s
2022-12-01T11:15:25.8544678Z 
2022-12-01T11:15:25.8544767Z OK
2022-12-01T11:15:25.8544830Z 
2022-12-01T11:15:25.8544959Z Generating XML reports...
2022-12-01T11:15:25.8545395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110619.xml
2022-12-01T11:15:25.8545763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8545938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8546319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8546510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8546530Z 
2022-12-01T11:15:25.8546636Z Running tests...
2022-12-01T11:15:25.8546899Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8547220Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8547482Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8547685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107480
2022-12-01T11:15:25.8547903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107481
2022-12-01T11:15:25.8548274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8548451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8548831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8549019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8549615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8549891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8550267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8550456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8550698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8550940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8551338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8551734Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8551961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8552196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8552437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8552655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8553049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8553439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8553540Z ok (3.799s)
2022-12-01T11:15:25.8553561Z 
2022-12-01T11:15:25.8553826Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8553938Z Ran 1 test in 3.799s
2022-12-01T11:15:25.8553958Z 
2022-12-01T11:15:25.8554049Z OK
2022-12-01T11:15:25.8554072Z 
2022-12-01T11:15:25.8554196Z Generating XML reports...
2022-12-01T11:15:25.8554721Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110625.xml
2022-12-01T11:15:25.8555098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8555273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8555652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8555841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8555861Z 
2022-12-01T11:15:25.8555968Z Running tests...
2022-12-01T11:15:25.8556231Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8556546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8556827Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8557578Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.702s)
2022-12-01T11:15:25.8557600Z 
2022-12-01T11:15:25.8557862Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8557957Z Ran 1 test in 1.703s
2022-12-01T11:15:25.8557977Z 
2022-12-01T11:15:25.8558082Z OK (skipped=1)
2022-12-01T11:15:25.8558101Z 
2022-12-01T11:15:25.8558225Z Generating XML reports...
2022-12-01T11:15:25.8558672Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110632.xml
2022-12-01T11:15:25.8559044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8559278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8559661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8559850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8559870Z 
2022-12-01T11:15:25.8559959Z Running tests...
2022-12-01T11:15:25.8560223Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8560535Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8560808Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8561547Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2022-12-01T11:15:25.8561570Z 
2022-12-01T11:15:25.8561835Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8561946Z Ran 1 test in 1.649s
2022-12-01T11:15:25.8561965Z 
2022-12-01T11:15:25.8562072Z OK (skipped=1)
2022-12-01T11:15:25.8562091Z 
2022-12-01T11:15:25.8562212Z Generating XML reports...
2022-12-01T11:15:25.8562664Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110635.xml
2022-12-01T11:15:25.8563019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8563196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8563572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8563812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8563834Z 
2022-12-01T11:15:25.8563945Z Running tests...
2022-12-01T11:15:25.8564207Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8564521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8564791Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8564997Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107847
2022-12-01T11:15:25.8565217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107848
2022-12-01T11:15:25.8565591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8565764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8566152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8566341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8566709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8566883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8567264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8567436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8567678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8567920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8568380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8568829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8569057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8569283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8569384Z ok (3.887s)
2022-12-01T11:15:25.8569404Z 
2022-12-01T11:15:25.8569671Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8569765Z Ran 1 test in 3.887s
2022-12-01T11:15:25.8569784Z 
2022-12-01T11:15:25.8569875Z OK
2022-12-01T11:15:25.8569894Z 
2022-12-01T11:15:25.8570016Z Generating XML reports...
2022-12-01T11:15:25.8570467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110639.xml
2022-12-01T11:15:25.8570846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8571023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8571402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8571592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8571611Z 
2022-12-01T11:15:25.8571701Z Running tests...
2022-12-01T11:15:25.8571963Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8572276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8572514Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8572737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108153
2022-12-01T11:15:25.8573004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108154
2022-12-01T11:15:25.8573384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8573549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8573926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8574101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8574470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8574644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8575024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8575217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8575461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8575702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8576099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8576479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8576706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8576931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8577270Z STAGE:2022-12-01 11:06:49 108153:108153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8577671Z STAGE:2022-12-01 11:06:49 108154:108154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8578008Z STAGE:2022-12-01 11:06:49 108154:108154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8578335Z STAGE:2022-12-01 11:06:49 108153:108153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8578665Z STAGE:2022-12-01 11:06:49 108154:108154 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8578987Z STAGE:2022-12-01 11:06:49 108153:108153 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8579308Z STAGE:2022-12-01 11:06:49 108153:108153 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8579644Z STAGE:2022-12-01 11:06:49 108154:108154 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8579745Z ok (3.879s)
2022-12-01T11:15:25.8579765Z 
2022-12-01T11:15:25.8580034Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8580147Z Ran 1 test in 3.879s
2022-12-01T11:15:25.8580167Z 
2022-12-01T11:15:25.8580257Z OK
2022-12-01T11:15:25.8580276Z 
2022-12-01T11:15:25.8580399Z Generating XML reports...
2022-12-01T11:15:25.8580854Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110646.xml
2022-12-01T11:15:25.8581228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8581386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8581766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8581955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8581975Z 
2022-12-01T11:15:25.8582083Z Running tests...
2022-12-01T11:15:25.8582344Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8582707Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8582967Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8583187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108464
2022-12-01T11:15:25.8583388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108465
2022-12-01T11:15:25.8583760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8583933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8584311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8584500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8584875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8585046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8585414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8585601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8585831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8586072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8586473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8586871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8587199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8587429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8587529Z ok (3.758s)
2022-12-01T11:15:25.8587548Z 
2022-12-01T11:15:25.8587814Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8587908Z Ran 1 test in 3.759s
2022-12-01T11:15:25.8587944Z 
2022-12-01T11:15:25.8588018Z OK
2022-12-01T11:15:25.8588037Z 
2022-12-01T11:15:25.8588160Z Generating XML reports...
2022-12-01T11:15:25.8588614Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110652.xml
2022-12-01T11:15:25.8588986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8589158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8589780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8589976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8589996Z 
2022-12-01T11:15:25.8590104Z Running tests...
2022-12-01T11:15:25.8590360Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8590672Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8590923Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T11:15:25.8590943Z 
2022-12-01T11:15:25.8591202Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8591313Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8591333Z 
2022-12-01T11:15:25.8591439Z OK (skipped=1)
2022-12-01T11:15:25.8591458Z 
2022-12-01T11:15:25.8591579Z Generating XML reports...
2022-12-01T11:15:25.8592104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110658.xml
2022-12-01T11:15:25.8592490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8592648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8593027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8593213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8593234Z 
2022-12-01T11:15:25.8593341Z Running tests...
2022-12-01T11:15:25.8593604Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8593915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8594171Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8594398Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108830
2022-12-01T11:15:25.8594600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108831
2022-12-01T11:15:25.8594972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8595146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8595522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8595712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8596079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8596254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8596710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8596899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8597125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8597364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8597761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8598155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8598382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8598606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8598852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8599090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8599485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8599859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8600194Z STAGE:2022-12-01 11:07:04 108831:108831 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8600518Z STAGE:2022-12-01 11:07:04 108830:108830 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8600852Z STAGE:2022-12-01 11:07:04 108831:108831 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8601190Z STAGE:2022-12-01 11:07:04 108830:108830 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8601569Z STAGE:2022-12-01 11:07:04 108831:108831 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8601907Z STAGE:2022-12-01 11:07:04 108830:108830 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8602240Z STAGE:2022-12-01 11:07:04 108830:108830 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8602576Z STAGE:2022-12-01 11:07:04 108831:108831 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8602661Z ok (4.207s)
2022-12-01T11:15:25.8602681Z 
2022-12-01T11:15:25.8602944Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8603055Z Ran 1 test in 4.207s
2022-12-01T11:15:25.8603075Z 
2022-12-01T11:15:25.8603165Z OK
2022-12-01T11:15:25.8603184Z 
2022-12-01T11:15:25.8603307Z Generating XML reports...
2022-12-01T11:15:25.8603762Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110700.xml
2022-12-01T11:15:25.8604145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8604317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8604679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8604867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8604887Z 
2022-12-01T11:15:25.8604992Z Running tests...
2022-12-01T11:15:25.8605254Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8605567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8605816Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8606090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109160
2022-12-01T11:15:25.8606311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109161
2022-12-01T11:15:25.8606687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8606843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8607220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8607408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8607774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8607946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8608321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8608515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8608759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8608982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8609378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8609776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8610007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8610232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8610390Z skip: Skipped due to small world size. (3.852s)
2022-12-01T11:15:25.8610414Z 
2022-12-01T11:15:25.8610726Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8610845Z Ran 1 test in 3.852s
2022-12-01T11:15:25.8610865Z 
2022-12-01T11:15:25.8610970Z OK (skipped=1)
2022-12-01T11:15:25.8610989Z 
2022-12-01T11:15:25.8611094Z Generating XML reports...
2022-12-01T11:15:25.8611546Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110706.xml
2022-12-01T11:15:25.8611921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8612095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8612470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8612658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8612682Z 
2022-12-01T11:15:25.8612789Z Running tests...
2022-12-01T11:15:25.8613054Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8613368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8613602Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8613820Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109431
2022-12-01T11:15:25.8614038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109432
2022-12-01T11:15:25.8614409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8614584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8614961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8615203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8615572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8615729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8616102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8616290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8616528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8616769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8617170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8617564Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8617801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8618028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8618111Z ok (3.948s)
2022-12-01T11:15:25.8618131Z 
2022-12-01T11:15:25.8618397Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8618509Z Ran 1 test in 3.948s
2022-12-01T11:15:25.8618528Z 
2022-12-01T11:15:25.8618619Z OK
2022-12-01T11:15:25.8618637Z 
2022-12-01T11:15:25.8618759Z Generating XML reports...
2022-12-01T11:15:25.8619209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110712.xml
2022-12-01T11:15:25.8619579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8619759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8620167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8620364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8620384Z 
2022-12-01T11:15:25.8620490Z Running tests...
2022-12-01T11:15:25.8620754Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8621064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8621328Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8622070Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.723s)
2022-12-01T11:15:25.8622094Z 
2022-12-01T11:15:25.8622360Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8622473Z Ran 1 test in 1.723s
2022-12-01T11:15:25.8622492Z 
2022-12-01T11:15:25.8622599Z OK (skipped=1)
2022-12-01T11:15:25.8622618Z 
2022-12-01T11:15:25.8622724Z Generating XML reports...
2022-12-01T11:15:25.8623169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110718.xml
2022-12-01T11:15:25.8623540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8623714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8624097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8624285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8624353Z 
2022-12-01T11:15:25.8624464Z Running tests...
2022-12-01T11:15:25.8624730Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8625026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8625271Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8625489Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109857
2022-12-01T11:15:25.8625705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109858
2022-12-01T11:15:25.8626075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8626248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8626625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8626821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8627187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8627343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8627717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8627906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8628148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8628389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8628789Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8629235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8629719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8629953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8630178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8630417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8630829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8631225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8631326Z ok (4.176s)
2022-12-01T11:15:25.8631350Z 
2022-12-01T11:15:25.8631617Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8631729Z Ran 1 test in 4.176s
2022-12-01T11:15:25.8631748Z 
2022-12-01T11:15:25.8631839Z OK
2022-12-01T11:15:25.8631859Z 
2022-12-01T11:15:25.8631980Z Generating XML reports...
2022-12-01T11:15:25.8632417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110722.xml
2022-12-01T11:15:25.8632793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8632966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8633348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8633535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8633555Z 
2022-12-01T11:15:25.8633745Z Running tests...
2022-12-01T11:15:25.8634014Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8634327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8634556Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8634776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110122
2022-12-01T11:15:25.8634993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110123
2022-12-01T11:15:25.8635364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8635538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8635915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8636104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8636476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8636648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8637006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8637194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8637437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8637678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8638078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8638475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8638763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8638999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8639082Z ok (4.011s)
2022-12-01T11:15:25.8639119Z 
2022-12-01T11:15:25.8639374Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8639485Z Ran 1 test in 4.011s
2022-12-01T11:15:25.8639505Z 
2022-12-01T11:15:25.8639596Z OK
2022-12-01T11:15:25.8639615Z 
2022-12-01T11:15:25.8639737Z Generating XML reports...
2022-12-01T11:15:25.8640186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110729.xml
2022-12-01T11:15:25.8640557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8640732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8641122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8641296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8641315Z 
2022-12-01T11:15:25.8641421Z Running tests...
2022-12-01T11:15:25.8641683Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8641995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8642235Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8642453Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110400
2022-12-01T11:15:25.8642671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110401
2022-12-01T11:15:25.8643040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8643273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8643661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8643849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8644215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8644383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8644758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8644945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8645187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8645436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8645821Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8646216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8646444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8646671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8646773Z ok (3.985s)
2022-12-01T11:15:25.8646792Z 
2022-12-01T11:15:25.8647054Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8647166Z Ran 1 test in 3.986s
2022-12-01T11:15:25.8647186Z 
2022-12-01T11:15:25.8647275Z OK
2022-12-01T11:15:25.8647295Z 
2022-12-01T11:15:25.8647400Z Generating XML reports...
2022-12-01T11:15:25.8647903Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110735.xml
2022-12-01T11:15:25.8648288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8648462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8648834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8649021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8649041Z 
2022-12-01T11:15:25.8649149Z Running tests...
2022-12-01T11:15:25.8649412Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8649728Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8649979Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8650208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110660
2022-12-01T11:15:25.8650424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110661
2022-12-01T11:15:25.8650795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8650967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8651342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8651530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8651895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8652066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8652491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8652681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8652923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8653165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8653565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8653962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8654186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8654409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8654636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8654873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8655266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8655657Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8655757Z ok (4.073s)
2022-12-01T11:15:25.8655777Z 
2022-12-01T11:15:25.8656039Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8656149Z Ran 1 test in 4.073s
2022-12-01T11:15:25.8656169Z 
2022-12-01T11:15:25.8656258Z OK
2022-12-01T11:15:25.8656277Z 
2022-12-01T11:15:25.8656398Z Generating XML reports...
2022-12-01T11:15:25.8656833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110741.xml
2022-12-01T11:15:25.8657259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8657441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8657821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8658010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8658029Z 
2022-12-01T11:15:25.8658137Z Running tests...
2022-12-01T11:15:25.8658398Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8658711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8658969Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8659180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 110992
2022-12-01T11:15:25.8659398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 110993
2022-12-01T11:15:25.8659771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8659943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8660322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8660511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8660877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8661052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8661415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8661662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8661906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8662148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8662547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8662940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8663164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8663387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8663625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8663851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8664244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8664636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8664736Z ok (4.004s)
2022-12-01T11:15:25.8664757Z 
2022-12-01T11:15:25.8665019Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8665133Z Ran 1 test in 4.004s
2022-12-01T11:15:25.8665153Z 
2022-12-01T11:15:25.8665243Z OK
2022-12-01T11:15:25.8665262Z 
2022-12-01T11:15:25.8665382Z Generating XML reports...
2022-12-01T11:15:25.8665833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110748.xml
2022-12-01T11:15:25.8666247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8666431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8666816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8667005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8667025Z 
2022-12-01T11:15:25.8667131Z Running tests...
2022-12-01T11:15:25.8667393Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8667704Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8667967Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8668171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111284
2022-12-01T11:15:25.8668391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111285
2022-12-01T11:15:25.8668821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8668994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8669651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8669853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8670227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8670401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8670777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8671038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8671286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8671530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8671931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8672322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8672550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8672777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8673034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfge9_ha3
2022-12-01T11:15:25.8673308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfge9_ha3/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8673548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwq7aie4v
2022-12-01T11:15:25.8673812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwq7aie4v/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8673912Z ok (6.160s)
2022-12-01T11:15:25.8673935Z 
2022-12-01T11:15:25.8674204Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8674315Z Ran 1 test in 6.160s
2022-12-01T11:15:25.8674337Z 
2022-12-01T11:15:25.8674456Z OK
2022-12-01T11:15:25.8674482Z 
2022-12-01T11:15:25.8674703Z Generating XML reports...
2022-12-01T11:15:25.8675223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110754.xml
2022-12-01T11:15:25.8675579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8675759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8676212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8676415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8676436Z 
2022-12-01T11:15:25.8676543Z Running tests...
2022-12-01T11:15:25.8676923Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8677243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8677478Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8677692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111589
2022-12-01T11:15:25.8677893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111590
2022-12-01T11:15:25.8678262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8678442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8678819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8679006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8679372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8679544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8679922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8680093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8680336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8680647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8681047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8681442Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8681671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8681897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8681997Z ok (4.079s)
2022-12-01T11:15:25.8682017Z 
2022-12-01T11:15:25.8682286Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8682381Z Ran 1 test in 4.079s
2022-12-01T11:15:25.8682401Z 
2022-12-01T11:15:25.8682487Z OK
2022-12-01T11:15:25.8682506Z 
2022-12-01T11:15:25.8682631Z Generating XML reports...
2022-12-01T11:15:25.8683087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110802.xml
2022-12-01T11:15:25.8683457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8683631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8684007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8684196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8684215Z 
2022-12-01T11:15:25.8684305Z Running tests...
2022-12-01T11:15:25.8684568Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8684881Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8685125Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8685388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 111886
2022-12-01T11:15:25.8685612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 111887
2022-12-01T11:15:25.8685983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8686156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8686534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8686708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8687076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8687248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8687631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8687825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8688067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8688310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8688710Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8689089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8689318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8689545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8689701Z ok (4.218s)
2022-12-01T11:15:25.8689721Z 
2022-12-01T11:15:25.8689992Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8690106Z Ran 1 test in 4.218s
2022-12-01T11:15:25.8690125Z 
2022-12-01T11:15:25.8690217Z OK
2022-12-01T11:15:25.8690236Z 
2022-12-01T11:15:25.8690358Z Generating XML reports...
2022-12-01T11:15:25.8690812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110809.xml
2022-12-01T11:15:25.8691168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8691341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8691714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8691903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8691927Z 
2022-12-01T11:15:25.8692035Z Running tests...
2022-12-01T11:15:25.8692302Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8692618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8692890Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8693093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112145
2022-12-01T11:15:25.8693308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112146
2022-12-01T11:15:25.8693680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8693855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8694231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8694470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8694848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8695021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8695398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8695571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8695818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8696061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8696461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8696865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8697097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8697323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8697664Z STAGE:2022-12-01 11:08:19 112146:112146 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8697994Z STAGE:2022-12-01 11:08:19 112145:112145 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8698315Z STAGE:2022-12-01 11:08:19 112146:112146 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8698651Z STAGE:2022-12-01 11:08:19 112145:112145 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8698752Z ok (4.081s)
2022-12-01T11:15:25.8698772Z 
2022-12-01T11:15:25.8699090Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8699207Z Ran 1 test in 4.081s
2022-12-01T11:15:25.8699227Z 
2022-12-01T11:15:25.8699316Z OK
2022-12-01T11:15:25.8699335Z 
2022-12-01T11:15:25.8699457Z Generating XML reports...
2022-12-01T11:15:25.8699908Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110815.xml
2022-12-01T11:15:25.8700264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8700440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8700819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8701009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8701028Z 
2022-12-01T11:15:25.8701136Z Running tests...
2022-12-01T11:15:25.8701406Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8701726Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8701991Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8702213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112446
2022-12-01T11:15:25.8702412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112447
2022-12-01T11:15:25.8702788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8702964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8703340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8703529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8703943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8704121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8704499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8704672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8704912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8705153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8705550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8705939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8706172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8706397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8706736Z STAGE:2022-12-01 11:08:25 112447:112447 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8707054Z STAGE:2022-12-01 11:08:25 112446:112446 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8707374Z STAGE:2022-12-01 11:08:25 112447:112447 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8707705Z STAGE:2022-12-01 11:08:25 112446:112446 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8707798Z ok (4.087s)
2022-12-01T11:15:25.8707819Z 
2022-12-01T11:15:25.8708080Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8708185Z Ran 1 test in 4.087s
2022-12-01T11:15:25.8708265Z 
2022-12-01T11:15:25.8708361Z OK
2022-12-01T11:15:25.8708379Z 
2022-12-01T11:15:25.8708508Z Generating XML reports...
2022-12-01T11:15:25.8708967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110822.xml
2022-12-01T11:15:25.8709336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8709685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8710073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8710263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8710284Z 
2022-12-01T11:15:25.8710393Z Running tests...
2022-12-01T11:15:25.8710657Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8710967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8711257Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8711478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 112672
2022-12-01T11:15:25.8711679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 112673
2022-12-01T11:15:25.8712052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8712230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8712606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8712795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8713163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8713410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8713802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8713993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8714219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8714461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8714860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8715254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8715482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8715713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8715957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8716197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8716595Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8716973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8717213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.8717450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.8717918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8718310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8718540Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:15:25.8718643Z ok (22.897s)
2022-12-01T11:15:25.8718663Z 
2022-12-01T11:15:25.8718929Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8719043Z Ran 1 test in 22.897s
2022-12-01T11:15:25.8719062Z 
2022-12-01T11:15:25.8719136Z OK
2022-12-01T11:15:25.8719154Z 
2022-12-01T11:15:25.8719279Z Generating XML reports...
2022-12-01T11:15:25.8719733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110828.xml
2022-12-01T11:15:25.8720107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8720286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8720671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8720861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8720881Z 
2022-12-01T11:15:25.8720987Z Running tests...
2022-12-01T11:15:25.8721234Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8721552Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8721849Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8722069Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113282
2022-12-01T11:15:25.8722284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113283
2022-12-01T11:15:25.8722700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8722882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8723261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8723451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8723802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8723981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8724356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8724544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8724796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8725037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8725435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8725831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8726058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8726269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8726507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8726744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8727199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8727590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8727829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.8728064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.8728450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8728836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.8729050Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:15:25.8729284Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:15:25.8729388Z ok (22.375s)
2022-12-01T11:15:25.8729408Z 
2022-12-01T11:15:25.8729672Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8729785Z Ran 1 test in 22.375s
2022-12-01T11:15:25.8729804Z 
2022-12-01T11:15:25.8729894Z OK
2022-12-01T11:15:25.8729914Z 
2022-12-01T11:15:25.8730036Z Generating XML reports...
2022-12-01T11:15:25.8730487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110853.xml
2022-12-01T11:15:25.8730842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8731016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8731398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8731594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8731697Z 
2022-12-01T11:15:25.8731814Z Running tests...
2022-12-01T11:15:25.8732081Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8732393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8732669Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8732887Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113726
2022-12-01T11:15:25.8733087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113727
2022-12-01T11:15:25.8733455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8733628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8734013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8734200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8734564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8734736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8735110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8735282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8735522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8735758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8736158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8736641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8736867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8737092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8737247Z skip: Skipped due to small world size. (3.839s)
2022-12-01T11:15:25.8737268Z 
2022-12-01T11:15:25.8737532Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8737625Z Ran 1 test in 3.839s
2022-12-01T11:15:25.8737661Z 
2022-12-01T11:15:25.8737750Z OK (skipped=1)
2022-12-01T11:15:25.8737768Z 
2022-12-01T11:15:25.8737891Z Generating XML reports...
2022-12-01T11:15:25.8738342Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110918.xml
2022-12-01T11:15:25.8738717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8738890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8739267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8739453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8739473Z 
2022-12-01T11:15:25.8739577Z Running tests...
2022-12-01T11:15:25.8739825Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8740133Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8740396Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8740616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 113942
2022-12-01T11:15:25.8740888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 113943
2022-12-01T11:15:25.8741276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8741448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8741827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8741999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8742367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8742541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8742919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8743114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8743357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8743596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8743996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8744394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8744608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8744836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8745064Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2022-12-01T11:15:25.8745216Z ok (5.854s)
2022-12-01T11:15:25.8745237Z 
2022-12-01T11:15:25.8745506Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8745616Z Ran 1 test in 5.854s
2022-12-01T11:15:25.8745635Z 
2022-12-01T11:15:25.8745726Z OK
2022-12-01T11:15:25.8745745Z 
2022-12-01T11:15:25.8745869Z Generating XML reports...
2022-12-01T11:15:25.8746319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110924.xml
2022-12-01T11:15:25.8746675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8746850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8747231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8747422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8747445Z 
2022-12-01T11:15:25.8747551Z Running tests...
2022-12-01T11:15:25.8747819Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8748135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8748412Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8748616Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114101
2022-12-01T11:15:25.8748833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114102
2022-12-01T11:15:25.8749202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8749554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8749950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8750210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8750592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8750762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8751137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8751307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8751547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8751789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8752187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8752592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8752820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8753048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8753285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8753520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8753902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8754296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8754511Z [E ProcessGroupGloo.cpp:137] Rank 0 timed out in monitoredBarrier after 0 ms.
2022-12-01T11:15:25.8754759Z No ranks successfully processed in monitoredBarrier.
2022-12-01T11:15:25.8754980Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2022-12-01T11:15:25.8755081Z ok (3.980s)
2022-12-01T11:15:25.8755102Z 
2022-12-01T11:15:25.8755369Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8755480Z Ran 1 test in 3.980s
2022-12-01T11:15:25.8755500Z 
2022-12-01T11:15:25.8755574Z OK
2022-12-01T11:15:25.8755593Z 
2022-12-01T11:15:25.8755716Z Generating XML reports...
2022-12-01T11:15:25.8756169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110931.xml
2022-12-01T11:15:25.8756539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8756712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8757099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8757291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8757310Z 
2022-12-01T11:15:25.8757418Z Running tests...
2022-12-01T11:15:25.8757679Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8757975Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8758251Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8758467Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114326
2022-12-01T11:15:25.8758680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114327
2022-12-01T11:15:25.8759046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8759222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8759642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8759842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8760197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8760369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8760745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8760933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8761173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8761412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8761819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8762213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8762440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8762648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8762886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8763126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8763525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8763977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8764207Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2022-12-01T11:15:25.8764309Z ok (4.095s)
2022-12-01T11:15:25.8764329Z 
2022-12-01T11:15:25.8764592Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8764703Z Ran 1 test in 4.095s
2022-12-01T11:15:25.8764722Z 
2022-12-01T11:15:25.8764796Z OK
2022-12-01T11:15:25.8764814Z 
2022-12-01T11:15:25.8764937Z Generating XML reports...
2022-12-01T11:15:25.8765387Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110938.xml
2022-12-01T11:15:25.8765760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8765934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8766322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8766509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8766529Z 
2022-12-01T11:15:25.8766637Z Running tests...
2022-12-01T11:15:25.8766885Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8767196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8767474Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8767693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 114544
2022-12-01T11:15:25.8767910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 114545
2022-12-01T11:15:25.8768276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8768498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8768942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8769133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8769489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8769663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8770034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8770221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8770462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8770711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8771111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8771503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8771732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8771944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8772102Z skip: Skipped due to small world size. (3.938s)
2022-12-01T11:15:25.8772122Z 
2022-12-01T11:15:25.8772389Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8772498Z Ran 1 test in 3.938s
2022-12-01T11:15:25.8772517Z 
2022-12-01T11:15:25.8772649Z OK (skipped=1)
2022-12-01T11:15:25.8772732Z 
2022-12-01T11:15:25.8772857Z Generating XML reports...
2022-12-01T11:15:25.8773313Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110944.xml
2022-12-01T11:15:25.8773682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8773840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8774217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8774400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8774420Z 
2022-12-01T11:15:25.8774526Z Running tests...
2022-12-01T11:15:25.8774788Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8775099Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8775508Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T11:15:25.8775529Z 
2022-12-01T11:15:25.8775790Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8775902Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8775921Z 
2022-12-01T11:15:25.8776010Z OK (skipped=1)
2022-12-01T11:15:25.8776029Z 
2022-12-01T11:15:25.8776151Z Generating XML reports...
2022-12-01T11:15:25.8776600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110950.xml
2022-12-01T11:15:25.8776971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8777145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8777525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8777719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8777783Z 
2022-12-01T11:15:25.8777896Z Running tests...
2022-12-01T11:15:25.8778161Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8778457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8778858Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T11:15:25.8778878Z 
2022-12-01T11:15:25.8779136Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8779245Z Ran 1 test in 0.003s
2022-12-01T11:15:25.8779264Z 
2022-12-01T11:15:25.8779370Z OK (skipped=1)
2022-12-01T11:15:25.8779390Z 
2022-12-01T11:15:25.8779508Z Generating XML reports...
2022-12-01T11:15:25.8779955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110952.xml
2022-12-01T11:15:25.8780332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8780506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8780869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8781058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8781078Z 
2022-12-01T11:15:25.8781182Z Running tests...
2022-12-01T11:15:25.8781443Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8781757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8782153Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2022-12-01T11:15:25.8782173Z 
2022-12-01T11:15:25.8782487Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8782596Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8782615Z 
2022-12-01T11:15:25.8782704Z OK (skipped=1)
2022-12-01T11:15:25.8782740Z 
2022-12-01T11:15:25.8782845Z Generating XML reports...
2022-12-01T11:15:25.8783293Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110954.xml
2022-12-01T11:15:25.8783662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8783834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8784214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8784403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8784423Z 
2022-12-01T11:15:25.8784527Z Running tests...
2022-12-01T11:15:25.8784791Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8785094Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8785488Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2022-12-01T11:15:25.8785508Z 
2022-12-01T11:15:25.8785768Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8785879Z Ran 1 test in 0.003s
2022-12-01T11:15:25.8785898Z 
2022-12-01T11:15:25.8786002Z OK (skipped=1)
2022-12-01T11:15:25.8786021Z 
2022-12-01T11:15:25.8786140Z Generating XML reports...
2022-12-01T11:15:25.8786584Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110956.xml
2022-12-01T11:15:25.8786952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8787129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8787539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8787734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8787754Z 
2022-12-01T11:15:25.8787859Z Running tests...
2022-12-01T11:15:25.8788127Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8788438Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8788729Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2022-12-01T11:15:25.8788749Z 
2022-12-01T11:15:25.8789008Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8789117Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8789136Z 
2022-12-01T11:15:25.8789246Z OK (skipped=1)
2022-12-01T11:15:25.8789266Z 
2022-12-01T11:15:25.8789544Z Generating XML reports...
2022-12-01T11:15:25.8790007Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110958.xml
2022-12-01T11:15:25.8790378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8790551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8790928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8791115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8791135Z 
2022-12-01T11:15:25.8791241Z Running tests...
2022-12-01T11:15:25.8791498Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8791797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8792133Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:15:25.8792152Z 
2022-12-01T11:15:25.8792413Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8792521Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8792539Z 
2022-12-01T11:15:25.8792644Z OK (skipped=1)
2022-12-01T11:15:25.8792663Z 
2022-12-01T11:15:25.8792779Z Generating XML reports...
2022-12-01T11:15:25.8793224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111000.xml
2022-12-01T11:15:25.8793593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8793761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8794123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8794317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8794337Z 
2022-12-01T11:15:25.8794442Z Running tests...
2022-12-01T11:15:25.8794701Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8795009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8795275Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:15:25.8795294Z 
2022-12-01T11:15:25.8795556Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8795666Z Ran 1 test in 0.003s
2022-12-01T11:15:25.8795685Z 
2022-12-01T11:15:25.8795789Z OK (skipped=1)
2022-12-01T11:15:25.8795808Z 
2022-12-01T11:15:25.8795913Z Generating XML reports...
2022-12-01T11:15:25.8796356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111002.xml
2022-12-01T11:15:25.8796785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8796967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8797346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8797531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8797551Z 
2022-12-01T11:15:25.8797656Z Running tests...
2022-12-01T11:15:25.8797915Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8798209Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8798518Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:15:25.8798542Z 
2022-12-01T11:15:25.8798801Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8798913Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8798933Z 
2022-12-01T11:15:25.8799036Z OK (skipped=1)
2022-12-01T11:15:25.8799055Z 
2022-12-01T11:15:25.8799175Z Generating XML reports...
2022-12-01T11:15:25.8799625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111004.xml
2022-12-01T11:15:25.8799996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8800165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8800526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8800717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8800736Z 
2022-12-01T11:15:25.8800898Z Running tests...
2022-12-01T11:15:25.8801163Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8801475Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8801772Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8801990Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115217
2022-12-01T11:15:25.8802209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115218
2022-12-01T11:15:25.8802577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8802734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8803105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8803296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8803664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8803835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8804210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8804399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8804643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8804882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8805266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8805663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8805932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8806163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8806264Z ok (3.859s)
2022-12-01T11:15:25.8806285Z 
2022-12-01T11:15:25.8806553Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8806664Z Ran 1 test in 3.859s
2022-12-01T11:15:25.8806683Z 
2022-12-01T11:15:25.8806774Z OK
2022-12-01T11:15:25.8806793Z 
2022-12-01T11:15:25.8806898Z Generating XML reports...
2022-12-01T11:15:25.8807346Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111006.xml
2022-12-01T11:15:25.8807722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8807895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8808273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8808458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8808477Z 
2022-12-01T11:15:25.8808583Z Running tests...
2022-12-01T11:15:25.8808842Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8809152Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8809427Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8809645Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115416
2022-12-01T11:15:25.8809863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115417
2022-12-01T11:15:25.8810299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8810468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8810845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8811031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8811396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8811553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8811929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8812120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8812366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8812613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8813013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8813409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8813634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8813858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8813942Z ok (3.833s)
2022-12-01T11:15:25.8813962Z 
2022-12-01T11:15:25.8814224Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8814334Z Ran 1 test in 3.833s
2022-12-01T11:15:25.8814353Z 
2022-12-01T11:15:25.8814449Z OK
2022-12-01T11:15:25.8814468Z 
2022-12-01T11:15:25.8814589Z Generating XML reports...
2022-12-01T11:15:25.8815079Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111012.xml
2022-12-01T11:15:25.8815459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8815630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8816005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8816177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8816197Z 
2022-12-01T11:15:25.8816299Z Running tests...
2022-12-01T11:15:25.8816556Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8816865Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8817143Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:15:25.8817162Z 
2022-12-01T11:15:25.8817418Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8817530Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8817550Z 
2022-12-01T11:15:25.8817653Z OK (skipped=1)
2022-12-01T11:15:25.8817672Z 
2022-12-01T11:15:25.8817777Z Generating XML reports...
2022-12-01T11:15:25.8818223Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111018.xml
2022-12-01T11:15:25.8818592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8818763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8819132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8819376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8819396Z 
2022-12-01T11:15:25.8819500Z Running tests...
2022-12-01T11:15:25.8819760Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8820064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8820344Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2022-12-01T11:15:25.8820378Z 
2022-12-01T11:15:25.8820619Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8820732Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8820751Z 
2022-12-01T11:15:25.8820857Z OK (skipped=1)
2022-12-01T11:15:25.8820876Z 
2022-12-01T11:15:25.8820993Z Generating XML reports...
2022-12-01T11:15:25.8821439Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111020.xml
2022-12-01T11:15:25.8821814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8821989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8822367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8822595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8822631Z 
2022-12-01T11:15:25.8822723Z Running tests...
2022-12-01T11:15:25.8822984Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8823393Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8823665Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8824463Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.719s)
2022-12-01T11:15:25.8824485Z 
2022-12-01T11:15:25.8824765Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8824878Z Ran 1 test in 1.719s
2022-12-01T11:15:25.8824897Z 
2022-12-01T11:15:25.8825001Z OK (skipped=1)
2022-12-01T11:15:25.8825020Z 
2022-12-01T11:15:25.8825142Z Generating XML reports...
2022-12-01T11:15:25.8825576Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111022.xml
2022-12-01T11:15:25.8825943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8826115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8826500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8826689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8826708Z 
2022-12-01T11:15:25.8826815Z Running tests...
2022-12-01T11:15:25.8827078Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8827395Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8827660Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8827882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 115815
2022-12-01T11:15:25.8828098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 115816
2022-12-01T11:15:25.8828464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8828691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8829071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8829260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8829800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8829974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8830336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8830525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8830768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8831011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8831407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8831798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8832025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8832252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8832506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3_8047r
2022-12-01T11:15:25.8832755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3_8047r/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8833005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0b2dob69
2022-12-01T11:15:25.8833337Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0b2dob69/_remote_module_non_scriptable.py
2022-12-01T11:15:25.8833450Z ok (5.842s)
2022-12-01T11:15:25.8833471Z 
2022-12-01T11:15:25.8833734Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8833843Z Ran 1 test in 5.842s
2022-12-01T11:15:25.8833864Z 
2022-12-01T11:15:25.8833952Z OK
2022-12-01T11:15:25.8833971Z 
2022-12-01T11:15:25.8834092Z Generating XML reports...
2022-12-01T11:15:25.8834528Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111026.xml
2022-12-01T11:15:25.8834902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8835072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8835451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8835644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8835668Z 
2022-12-01T11:15:25.8835775Z Running tests...
2022-12-01T11:15:25.8836034Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8836345Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8836616Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8836819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116062
2022-12-01T11:15:25.8837034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116063
2022-12-01T11:15:25.8837406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8837577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8838041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8838230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8838598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8838771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8839125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8839311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8839549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8839793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8840199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8840596Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8840821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8841043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8841142Z ok (6.413s)
2022-12-01T11:15:25.8841162Z 
2022-12-01T11:15:25.8841409Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8841519Z Ran 1 test in 6.413s
2022-12-01T11:15:25.8841538Z 
2022-12-01T11:15:25.8841628Z OK
2022-12-01T11:15:25.8841647Z 
2022-12-01T11:15:25.8841766Z Generating XML reports...
2022-12-01T11:15:25.8842211Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111034.xml
2022-12-01T11:15:25.8842631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8842814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8843191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8843363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8843396Z 
2022-12-01T11:15:25.8843485Z Running tests...
2022-12-01T11:15:25.8843750Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8844061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8844347Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8844567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116277
2022-12-01T11:15:25.8844792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116278
2022-12-01T11:15:25.8845161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8845334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8845695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8845886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8846256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8846429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8846802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8847040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8847284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8847523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8847922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8848305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8848538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8848758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8848858Z ok (6.689s)
2022-12-01T11:15:25.8848878Z 
2022-12-01T11:15:25.8849141Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8849255Z Ran 1 test in 6.689s
2022-12-01T11:15:25.8849277Z 
2022-12-01T11:15:25.8849367Z OK
2022-12-01T11:15:25.8849387Z 
2022-12-01T11:15:25.8849509Z Generating XML reports...
2022-12-01T11:15:25.8849947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111042.xml
2022-12-01T11:15:25.8850312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8850481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8850850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8851036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8851056Z 
2022-12-01T11:15:25.8851161Z Running tests...
2022-12-01T11:15:25.8851424Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8851792Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8852075Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8852808Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.579s)
2022-12-01T11:15:25.8852845Z 
2022-12-01T11:15:25.8853091Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8853202Z Ran 1 test in 1.579s
2022-12-01T11:15:25.8853222Z 
2022-12-01T11:15:25.8853328Z OK (skipped=1)
2022-12-01T11:15:25.8853347Z 
2022-12-01T11:15:25.8853465Z Generating XML reports...
2022-12-01T11:15:25.8853910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111051.xml
2022-12-01T11:15:25.8854280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8854451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8854824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8854998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8855034Z 
2022-12-01T11:15:25.8855123Z Running tests...
2022-12-01T11:15:25.8855380Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8855693Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8855984Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8856779Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.661s)
2022-12-01T11:15:25.8856801Z 
2022-12-01T11:15:25.8857059Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8857169Z Ran 1 test in 1.661s
2022-12-01T11:15:25.8857189Z 
2022-12-01T11:15:25.8857293Z OK (skipped=1)
2022-12-01T11:15:25.8857312Z 
2022-12-01T11:15:25.8857433Z Generating XML reports...
2022-12-01T11:15:25.8857864Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111055.xml
2022-12-01T11:15:25.8858235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8858411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8858790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8858978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8858998Z 
2022-12-01T11:15:25.8859101Z Running tests...
2022-12-01T11:15:25.8859361Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8859676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8859974Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8860180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116697
2022-12-01T11:15:25.8860396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116698
2022-12-01T11:15:25.8860817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8860995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8861375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8861562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8861928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8862100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8862462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8862653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8862893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8863143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8863541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8863937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8864162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8864385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8864534Z skip: Need at least 4 CUDA devices (3.774s)
2022-12-01T11:15:25.8864554Z 
2022-12-01T11:15:25.8864801Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8864913Z Ran 1 test in 3.774s
2022-12-01T11:15:25.8864978Z 
2022-12-01T11:15:25.8865094Z OK (skipped=1)
2022-12-01T11:15:25.8865112Z 
2022-12-01T11:15:25.8865238Z Generating XML reports...
2022-12-01T11:15:25.8865689Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111058.xml
2022-12-01T11:15:25.8866062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8866235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8866609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8866795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8866815Z 
2022-12-01T11:15:25.8866905Z Running tests...
2022-12-01T11:15:25.8867165Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8867478Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8867805Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8868023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 116924
2022-12-01T11:15:25.8868233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 116925
2022-12-01T11:15:25.8868646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8868822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8869185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8869532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8869918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8870165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8870556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8870743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8870981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8871223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8871615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8871997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8872224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8872454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8872604Z skip: Need at least 4 CUDA devices (3.866s)
2022-12-01T11:15:25.8872626Z 
2022-12-01T11:15:25.8872892Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8873004Z Ran 1 test in 3.867s
2022-12-01T11:15:25.8873023Z 
2022-12-01T11:15:25.8873123Z OK (skipped=1)
2022-12-01T11:15:25.8873143Z 
2022-12-01T11:15:25.8873264Z Generating XML reports...
2022-12-01T11:15:25.8873879Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111104.xml
2022-12-01T11:15:25.8874245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8874416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8874794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8875104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8875125Z 
2022-12-01T11:15:25.8875235Z Running tests...
2022-12-01T11:15:25.8875498Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8875805Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8876087Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8876836Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.618s)
2022-12-01T11:15:25.8876861Z 
2022-12-01T11:15:25.8877123Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8877222Z Ran 1 test in 1.619s
2022-12-01T11:15:25.8877241Z 
2022-12-01T11:15:25.8877344Z OK (skipped=1)
2022-12-01T11:15:25.8877363Z 
2022-12-01T11:15:25.8877588Z Generating XML reports...
2022-12-01T11:15:25.8878050Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111110.xml
2022-12-01T11:15:25.8878423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8878593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8878962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8879148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8879168Z 
2022-12-01T11:15:25.8879259Z Running tests...
2022-12-01T11:15:25.8879522Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8879887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8880155Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8880372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117231
2022-12-01T11:15:25.8880590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117232
2022-12-01T11:15:25.8880958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8881127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8881499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8881672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8882045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8882219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8882593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8882780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8883013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8883245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8883636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8884017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8884296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8884511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8884745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8884973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8885360Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8885752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8886094Z STAGE:2022-12-01 11:11:18 117231:117231 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8886429Z STAGE:2022-12-01 11:11:18 117232:117232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8886753Z STAGE:2022-12-01 11:11:18 117232:117232 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8887087Z STAGE:2022-12-01 11:11:18 117231:117231 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8887419Z STAGE:2022-12-01 11:11:18 117232:117232 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8887738Z STAGE:2022-12-01 11:11:18 117231:117231 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8888285Z STAGE:2022-12-01 11:11:18 117232:117232 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:11:18 117231:117231 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8888307Z 
2022-12-01T11:15:25.8888407Z ok (3.835s)
2022-12-01T11:15:25.8888426Z 
2022-12-01T11:15:25.8888686Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8888798Z Ran 1 test in 3.836s
2022-12-01T11:15:25.8888863Z 
2022-12-01T11:15:25.8888954Z OK
2022-12-01T11:15:25.8888973Z 
2022-12-01T11:15:25.8889093Z Generating XML reports...
2022-12-01T11:15:25.8889532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111114.xml
2022-12-01T11:15:25.8889903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8890075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8890453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8890638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8890658Z 
2022-12-01T11:15:25.8890762Z Running tests...
2022-12-01T11:15:25.8891025Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8891348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8891599Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8891815Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117452
2022-12-01T11:15:25.8892027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117453
2022-12-01T11:15:25.8892397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8892566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8892927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8893099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8893534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8893720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8894080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8894263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8894504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8894741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8895139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8895533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8895769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8895998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8896235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8896458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8896854Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8897246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8897582Z STAGE:2022-12-01 11:11:24 117452:117452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8897909Z STAGE:2022-12-01 11:11:24 117453:117453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8898507Z STAGE:2022-12-01 11:11:24 117453:117453 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:11:24 117452:117452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8898530Z 
2022-12-01T11:15:25.8898868Z STAGE:2022-12-01 11:11:24 117453:117453 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8899191Z STAGE:2022-12-01 11:11:24 117452:117452 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8899524Z STAGE:2022-12-01 11:11:24 117453:117453 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8899854Z STAGE:2022-12-01 11:11:24 117452:117452 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8899940Z ok (3.944s)
2022-12-01T11:15:25.8899959Z 
2022-12-01T11:15:25.8900219Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8900335Z Ran 1 test in 3.944s
2022-12-01T11:15:25.8900354Z 
2022-12-01T11:15:25.8900445Z OK
2022-12-01T11:15:25.8900467Z 
2022-12-01T11:15:25.8900589Z Generating XML reports...
2022-12-01T11:15:25.8901043Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111120.xml
2022-12-01T11:15:25.8901412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8901585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8901950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8902138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8902158Z 
2022-12-01T11:15:25.8902262Z Running tests...
2022-12-01T11:15:25.8902524Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8902904Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8903178Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8903400Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117623
2022-12-01T11:15:25.8903615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117624
2022-12-01T11:15:25.8903972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8904145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8904516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8904706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8905071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8905249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8905625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8905814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8906054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8906280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8906683Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8907081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8907311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8907589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8907836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8908074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8908468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8908853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8909176Z STAGE:2022-12-01 11:11:30 117624:117624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8909688Z STAGE:2022-12-01 11:11:30 117623:117623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8910051Z STAGE:2022-12-01 11:11:30 117624:117624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8910387Z STAGE:2022-12-01 11:11:30 117623:117623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8910717Z STAGE:2022-12-01 11:11:30 117624:117624 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8911042Z STAGE:2022-12-01 11:11:30 117623:117623 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8911376Z STAGE:2022-12-01 11:11:30 117623:117623 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8911709Z STAGE:2022-12-01 11:11:30 117624:117624 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8911807Z ok (3.828s)
2022-12-01T11:15:25.8911827Z 
2022-12-01T11:15:25.8912075Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8912183Z Ran 1 test in 3.829s
2022-12-01T11:15:25.8912273Z 
2022-12-01T11:15:25.8912364Z OK
2022-12-01T11:15:25.8912383Z 
2022-12-01T11:15:25.8912513Z Generating XML reports...
2022-12-01T11:15:25.8912973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111126.xml
2022-12-01T11:15:25.8913340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8913515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8913892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8914066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8914099Z 
2022-12-01T11:15:25.8914190Z Running tests...
2022-12-01T11:15:25.8914448Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8914757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8915021Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8915240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 117848
2022-12-01T11:15:25.8915455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 117849
2022-12-01T11:15:25.8915824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8915995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8916355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8916542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8916911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8917088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8917537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8917739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8917980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8918227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8918629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8919008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8919236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8919468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8919703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.8919937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.8920331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8920722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.8921061Z STAGE:2022-12-01 11:11:36 117848:117848 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8921390Z STAGE:2022-12-01 11:11:36 117849:117849 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8921710Z STAGE:2022-12-01 11:11:36 117849:117849 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8922110Z STAGE:2022-12-01 11:11:36 117848:117848 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8922435Z STAGE:2022-12-01 11:11:36 117849:117849 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8922756Z STAGE:2022-12-01 11:11:36 117848:117848 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8923293Z STAGE:2022-12-01 11:11:36 117849:117849 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:11:36 117848:117848 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8923314Z 
2022-12-01T11:15:25.8923412Z ok (3.925s)
2022-12-01T11:15:25.8923430Z 
2022-12-01T11:15:25.8923689Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8923799Z Ran 1 test in 3.925s
2022-12-01T11:15:25.8923818Z 
2022-12-01T11:15:25.8923905Z OK
2022-12-01T11:15:25.8923927Z 
2022-12-01T11:15:25.8924033Z Generating XML reports...
2022-12-01T11:15:25.8924480Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111132.xml
2022-12-01T11:15:25.8924855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8925030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8925410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8925597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8925616Z 
2022-12-01T11:15:25.8925718Z Running tests...
2022-12-01T11:15:25.8925980Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8926292Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8926581Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8926808Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118009
2022-12-01T11:15:25.8927021Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118010
2022-12-01T11:15:25.8927391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8927562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8927932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8928119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8928486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8928643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8929029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8929217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8929462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8929695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8930092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8930488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8930717Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8930936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8931141Z skip: Skipped due to small world size. (3.839s)
2022-12-01T11:15:25.8931162Z 
2022-12-01T11:15:25.8931426Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8931537Z Ran 1 test in 3.839s
2022-12-01T11:15:25.8931557Z 
2022-12-01T11:15:25.8931662Z OK (skipped=1)
2022-12-01T11:15:25.8931681Z 
2022-12-01T11:15:25.8931804Z Generating XML reports...
2022-12-01T11:15:25.8932253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111138.xml
2022-12-01T11:15:25.8932621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8932797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8933174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8933354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8933373Z 
2022-12-01T11:15:25.8933476Z Running tests...
2022-12-01T11:15:25.8933735Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8934047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8934302Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8934519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118204
2022-12-01T11:15:25.8934733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118205
2022-12-01T11:15:25.8935105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8935263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8935689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8935883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8936255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8936427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8936803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8936992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8937231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8937469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8937854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8938257Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8938485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8938711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8938868Z skip: Skipped due to small world size. (4.196s)
2022-12-01T11:15:25.8938889Z 
2022-12-01T11:15:25.8939156Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8939264Z Ran 1 test in 4.197s
2022-12-01T11:15:25.8939284Z 
2022-12-01T11:15:25.8939388Z OK (skipped=1)
2022-12-01T11:15:25.8939407Z 
2022-12-01T11:15:25.8939511Z Generating XML reports...
2022-12-01T11:15:25.8939960Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111144.xml
2022-12-01T11:15:25.8940393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8940570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8940944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8941133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8941153Z 
2022-12-01T11:15:25.8941255Z Running tests...
2022-12-01T11:15:25.8941516Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8941824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8942069Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8942285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118427
2022-12-01T11:15:25.8942509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118428
2022-12-01T11:15:25.8942877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8943050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8943426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8943610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8943976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8944147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8944508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8944701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8944991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8945239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8945638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8946034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8946260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8946482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8946624Z skip: Skipped due to small world size. (3.857s)
2022-12-01T11:15:25.8946664Z 
2022-12-01T11:15:25.8946913Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8947026Z Ran 1 test in 3.857s
2022-12-01T11:15:25.8947046Z 
2022-12-01T11:15:25.8947152Z OK (skipped=1)
2022-12-01T11:15:25.8947171Z 
2022-12-01T11:15:25.8947292Z Generating XML reports...
2022-12-01T11:15:25.8947741Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111151.xml
2022-12-01T11:15:25.8948107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8948279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8948653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8948825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8948844Z 
2022-12-01T11:15:25.8948950Z Running tests...
2022-12-01T11:15:25.8949267Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8949911Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8950166Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8950382Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118575
2022-12-01T11:15:25.8950593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118576
2022-12-01T11:15:25.8950964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8951119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8951494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8951683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8952060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8952233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8952607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8952795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8953037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8953271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8953653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8954045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8954349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8954583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8954737Z skip: Skipped due to small world size. (5.418s)
2022-12-01T11:15:25.8954758Z 
2022-12-01T11:15:25.8955024Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8955134Z Ran 1 test in 5.418s
2022-12-01T11:15:25.8955153Z 
2022-12-01T11:15:25.8955257Z OK (skipped=1)
2022-12-01T11:15:25.8955276Z 
2022-12-01T11:15:25.8955397Z Generating XML reports...
2022-12-01T11:15:25.8955833Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111157.xml
2022-12-01T11:15:25.8956202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8956374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8956755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8956946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8956966Z 
2022-12-01T11:15:25.8957072Z Running tests...
2022-12-01T11:15:25.8957333Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8957645Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8957873Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8958089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118774
2022-12-01T11:15:25.8958304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118775
2022-12-01T11:15:25.8958677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8958923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8959306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8959494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8959856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8960029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8960389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8960578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8960816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8961067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8961466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8961858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8962082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8962419Z STAGE:2022-12-01 11:12:08 118775:118775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8962648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8962968Z STAGE:2022-12-01 11:12:08 118774:118774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8963565Z STAGE:2022-12-01 11:12:08 118774:118774 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:12:08 118775:118775 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8963590Z 
2022-12-01T11:15:25.8963934Z STAGE:2022-12-01 11:12:08 118775:118775 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8964255Z STAGE:2022-12-01 11:12:08 118774:118774 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8964800Z STAGE:2022-12-01 11:12:08 118775:118775 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:12:08 118774:118774 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8964820Z 
2022-12-01T11:15:25.8964918Z ok (3.883s)
2022-12-01T11:15:25.8964937Z 
2022-12-01T11:15:25.8965196Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8965305Z Ran 1 test in 3.883s
2022-12-01T11:15:25.8965324Z 
2022-12-01T11:15:25.8965418Z OK
2022-12-01T11:15:25.8965437Z 
2022-12-01T11:15:25.8965560Z Generating XML reports...
2022-12-01T11:15:25.8966003Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111204.xml
2022-12-01T11:15:25.8966377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8966549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8966929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8967117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8967137Z 
2022-12-01T11:15:25.8967244Z Running tests...
2022-12-01T11:15:25.8967507Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8967821Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8968122Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8968346Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 118887
2022-12-01T11:15:25.8968559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 118888
2022-12-01T11:15:25.8968980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8969151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8969531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8969714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8970072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8970243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8970614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8970803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8971043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8971288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8971689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8972086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8972310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8972535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8972927Z STAGE:2022-12-01 11:12:15 118887:118887 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8973250Z STAGE:2022-12-01 11:12:15 118888:118888 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8973583Z STAGE:2022-12-01 11:12:15 118888:118888 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8973917Z STAGE:2022-12-01 11:12:15 118887:118887 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8974246Z STAGE:2022-12-01 11:12:15 118888:118888 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8974563Z STAGE:2022-12-01 11:12:15 118887:118887 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8974899Z STAGE:2022-12-01 11:12:15 118887:118887 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8975239Z STAGE:2022-12-01 11:12:15 118888:118888 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8975336Z ok (4.657s)
2022-12-01T11:15:25.8975357Z 
2022-12-01T11:15:25.8975621Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8975716Z Ran 1 test in 4.658s
2022-12-01T11:15:25.8975735Z 
2022-12-01T11:15:25.8975818Z OK
2022-12-01T11:15:25.8975837Z 
2022-12-01T11:15:25.8975957Z Generating XML reports...
2022-12-01T11:15:25.8976415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111210.xml
2022-12-01T11:15:25.8976784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8976959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8977338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8977581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8977604Z 
2022-12-01T11:15:25.8977694Z Running tests...
2022-12-01T11:15:25.8977959Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8978272Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8978547Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2022-12-01T11:15:25.8978567Z 
2022-12-01T11:15:25.8978828Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8978933Z Ran 1 test in 0.002s
2022-12-01T11:15:25.8978952Z 
2022-12-01T11:15:25.8979055Z OK (skipped=1)
2022-12-01T11:15:25.8979074Z 
2022-12-01T11:15:25.8979194Z Generating XML reports...
2022-12-01T11:15:25.8979644Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111217.xml
2022-12-01T11:15:25.8980006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8980176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8980555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8980744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8980763Z 
2022-12-01T11:15:25.8980867Z Running tests...
2022-12-01T11:15:25.8981126Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8981435Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8981685Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8981889Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119033
2022-12-01T11:15:25.8982162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119034
2022-12-01T11:15:25.8982543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8982714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8983087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8983275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8983640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8983811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8984184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8984360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8984605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.8984847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.8985241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8985634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.8985861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.8986196Z STAGE:2022-12-01 11:12:22 119034:119034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8986425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.8986823Z STAGE:2022-12-01 11:12:22 119033:119033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8987145Z STAGE:2022-12-01 11:12:22 119033:119033 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8987478Z STAGE:2022-12-01 11:12:22 119034:119034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8987808Z STAGE:2022-12-01 11:12:22 119034:119034 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8988129Z STAGE:2022-12-01 11:12:22 119033:119033 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.8988678Z STAGE:2022-12-01 11:12:22 119033:119033 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:12:22 119034:119034 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.8988699Z 
2022-12-01T11:15:25.8988800Z ok (3.998s)
2022-12-01T11:15:25.8988819Z 
2022-12-01T11:15:25.8989079Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8989192Z Ran 1 test in 3.998s
2022-12-01T11:15:25.8989214Z 
2022-12-01T11:15:25.8989305Z OK
2022-12-01T11:15:25.8989324Z 
2022-12-01T11:15:25.8989737Z Generating XML reports...
2022-12-01T11:15:25.8990207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111219.xml
2022-12-01T11:15:25.8990576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8990751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8991126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8991314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8991334Z 
2022-12-01T11:15:25.8991440Z Running tests...
2022-12-01T11:15:25.8991702Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8992096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8992386Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2022-12-01T11:15:25.8992405Z 
2022-12-01T11:15:25.8992667Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8992776Z Ran 1 test in 0.003s
2022-12-01T11:15:25.8992795Z 
2022-12-01T11:15:25.8992900Z OK (skipped=1)
2022-12-01T11:15:25.8992918Z 
2022-12-01T11:15:25.8993038Z Generating XML reports...
2022-12-01T11:15:25.8993484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111225.xml
2022-12-01T11:15:25.8993855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8994025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8994414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8994589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8994609Z 
2022-12-01T11:15:25.8994715Z Running tests...
2022-12-01T11:15:25.8994974Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8995286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8995558Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2022-12-01T11:15:25.8995578Z 
2022-12-01T11:15:25.8995841Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8995952Z Ran 1 test in 0.003s
2022-12-01T11:15:25.8995971Z 
2022-12-01T11:15:25.8996077Z OK (skipped=1)
2022-12-01T11:15:25.8996155Z 
2022-12-01T11:15:25.8996267Z Generating XML reports...
2022-12-01T11:15:25.8996724Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111227.xml
2022-12-01T11:15:25.8997091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8997260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.8997637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.8997824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.8997844Z 
2022-12-01T11:15:25.8997957Z Running tests...
2022-12-01T11:15:25.8998216Z ----------------------------------------------------------------------
2022-12-01T11:15:25.8998531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.8998763Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.8998985Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119212
2022-12-01T11:15:25.8999206Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119213
2022-12-01T11:15:25.8999581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.8999751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9000124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9000309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9000671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9000829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9001254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9001451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9001695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9001938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9002338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9002731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9002961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9003186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9003520Z STAGE:2022-12-01 11:12:32 119213:119213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9003851Z STAGE:2022-12-01 11:12:32 119212:119212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9004181Z STAGE:2022-12-01 11:12:32 119213:119213 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9004519Z STAGE:2022-12-01 11:12:32 119212:119212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9004849Z STAGE:2022-12-01 11:12:32 119213:119213 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9005174Z STAGE:2022-12-01 11:12:32 119212:119212 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9005504Z STAGE:2022-12-01 11:12:32 119212:119212 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9005839Z STAGE:2022-12-01 11:12:32 119213:119213 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9005989Z ok (3.714s)
2022-12-01T11:15:25.9006009Z 
2022-12-01T11:15:25.9006258Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9006370Z Ran 1 test in 3.714s
2022-12-01T11:15:25.9006390Z 
2022-12-01T11:15:25.9006480Z OK
2022-12-01T11:15:25.9006499Z 
2022-12-01T11:15:25.9006620Z Generating XML reports...
2022-12-01T11:15:25.9007066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111229.xml
2022-12-01T11:15:25.9007434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9007609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9007985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9008162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9008196Z 
2022-12-01T11:15:25.9008289Z Running tests...
2022-12-01T11:15:25.9008552Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9008867Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9009123Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T11:15:25.9009143Z 
2022-12-01T11:15:25.9009399Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9009506Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9009526Z 
2022-12-01T11:15:25.9009628Z OK (skipped=1)
2022-12-01T11:15:25.9009648Z 
2022-12-01T11:15:25.9009766Z Generating XML reports...
2022-12-01T11:15:25.9010198Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111234.xml
2022-12-01T11:15:25.9010611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9010793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9011174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9011359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9011379Z 
2022-12-01T11:15:25.9011483Z Running tests...
2022-12-01T11:15:25.9011740Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9012046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9012309Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2022-12-01T11:15:25.9012329Z 
2022-12-01T11:15:25.9012574Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9012684Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9012703Z 
2022-12-01T11:15:25.9012805Z OK (skipped=1)
2022-12-01T11:15:25.9012824Z 
2022-12-01T11:15:25.9012944Z Generating XML reports...
2022-12-01T11:15:25.9013391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111236.xml
2022-12-01T11:15:25.9013759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9013932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9014307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9014497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9014516Z 
2022-12-01T11:15:25.9014607Z Running tests...
2022-12-01T11:15:25.9014862Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9015233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9015486Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9015704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119391
2022-12-01T11:15:25.9015920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119392
2022-12-01T11:15:25.9016289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9016455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9016817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9017002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9017368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9017545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9017927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9018108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9018344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9018585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9018981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9019361Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9019595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9020013Z STAGE:2022-12-01 11:12:42 119392:119392 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9020250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9020584Z STAGE:2022-12-01 11:12:42 119391:119391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9021134Z STAGE:2022-12-01 11:12:42 119392:119392 ActivityProfilerController.cpp:300] Completed Stage: CollectionSTAGE:2022-12-01 11:12:42 119391:119391 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9021155Z 
2022-12-01T11:15:25.9021483Z STAGE:2022-12-01 11:12:42 119392:119392 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9021808Z STAGE:2022-12-01 11:12:42 119391:119391 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9022147Z STAGE:2022-12-01 11:12:42 119391:119391 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9022468Z STAGE:2022-12-01 11:12:42 119392:119392 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9022565Z ok (3.936s)
2022-12-01T11:15:25.9022585Z 
2022-12-01T11:15:25.9022849Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9022959Z Ran 1 test in 3.936s
2022-12-01T11:15:25.9022979Z 
2022-12-01T11:15:25.9023064Z OK
2022-12-01T11:15:25.9023083Z 
2022-12-01T11:15:25.9023199Z Generating XML reports...
2022-12-01T11:15:25.9023646Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111238.xml
2022-12-01T11:15:25.9024014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9024186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9024608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9024796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9024816Z 
2022-12-01T11:15:25.9024924Z Running tests...
2022-12-01T11:15:25.9025185Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9025494Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9025730Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9025944Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119504
2022-12-01T11:15:25.9026161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119505
2022-12-01T11:15:25.9026512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9026690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9027070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9027264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9027629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9027793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9028172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9028358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9028596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9028820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9029269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9029918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9030150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9030373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9030714Z STAGE:2022-12-01 11:12:48 119504:119504 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9031038Z STAGE:2022-12-01 11:12:48 119505:119505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9031370Z STAGE:2022-12-01 11:12:48 119505:119505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9031712Z STAGE:2022-12-01 11:12:48 119504:119504 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9032032Z STAGE:2022-12-01 11:12:48 119504:119504 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9032352Z STAGE:2022-12-01 11:12:48 119505:119505 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9032686Z STAGE:2022-12-01 11:12:48 119505:119505 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9033018Z STAGE:2022-12-01 11:12:48 119504:119504 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9033119Z ok (3.672s)
2022-12-01T11:15:25.9033140Z 
2022-12-01T11:15:25.9033400Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9033509Z Ran 1 test in 3.672s
2022-12-01T11:15:25.9033529Z 
2022-12-01T11:15:25.9033610Z OK
2022-12-01T11:15:25.9033630Z 
2022-12-01T11:15:25.9033736Z Generating XML reports...
2022-12-01T11:15:25.9034296Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111244.xml
2022-12-01T11:15:25.9034668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9034843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9035215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9035402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9035422Z 
2022-12-01T11:15:25.9035523Z Running tests...
2022-12-01T11:15:25.9035784Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9036092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9036326Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9036555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119617
2022-12-01T11:15:25.9036773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119618
2022-12-01T11:15:25.9037145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9037316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9037691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9037876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9038240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9038395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9038764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9039014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9039267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9039509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9039907Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9040298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9040526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9040751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9040839Z ok (3.628s)
2022-12-01T11:15:25.9040860Z 
2022-12-01T11:15:25.9041123Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9041234Z Ran 1 test in 3.628s
2022-12-01T11:15:25.9041254Z 
2022-12-01T11:15:25.9041341Z OK
2022-12-01T11:15:25.9041360Z 
2022-12-01T11:15:25.9041482Z Generating XML reports...
2022-12-01T11:15:25.9041932Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111250.xml
2022-12-01T11:15:25.9042303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9042474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9042847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9043021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9043041Z 
2022-12-01T11:15:25.9043196Z Running tests...
2022-12-01T11:15:25.9043462Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9043774Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9044022Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9044239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119726
2022-12-01T11:15:25.9044450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119727
2022-12-01T11:15:25.9044812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9044968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9045340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9045526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9045896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9046068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9046444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9046628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9046863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9047101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9047485Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9047878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9048152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9048494Z STAGE:2022-12-01 11:12:59 119727:119727 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9048718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9049045Z STAGE:2022-12-01 11:12:59 119726:119726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9049380Z STAGE:2022-12-01 11:12:59 119726:119726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9049711Z STAGE:2022-12-01 11:12:59 119727:119727 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9050042Z STAGE:2022-12-01 11:12:59 119727:119727 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9050354Z STAGE:2022-12-01 11:12:59 119726:119726 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9050690Z STAGE:2022-12-01 11:12:59 119726:119726 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9051026Z STAGE:2022-12-01 11:12:59 119727:119727 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9051126Z ok (3.759s)
2022-12-01T11:15:25.9051147Z 
2022-12-01T11:15:25.9051409Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9051518Z Ran 1 test in 3.760s
2022-12-01T11:15:25.9051537Z 
2022-12-01T11:15:25.9051627Z OK
2022-12-01T11:15:25.9051646Z 
2022-12-01T11:15:25.9051764Z Generating XML reports...
2022-12-01T11:15:25.9052201Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111256.xml
2022-12-01T11:15:25.9052574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9052798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9053182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9053370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9053390Z 
2022-12-01T11:15:25.9053495Z Running tests...
2022-12-01T11:15:25.9053754Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9054064Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9054316Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T11:15:25.9054337Z 
2022-12-01T11:15:25.9054582Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9054694Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9054713Z 
2022-12-01T11:15:25.9054824Z OK (skipped=1)
2022-12-01T11:15:25.9054843Z 
2022-12-01T11:15:25.9054963Z Generating XML reports...
2022-12-01T11:15:25.9055417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111302.xml
2022-12-01T11:15:25.9055787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9055957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9056333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9056521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9056541Z 
2022-12-01T11:15:25.9056631Z Running tests...
2022-12-01T11:15:25.9056894Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9057204Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9057517Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2022-12-01T11:15:25.9057538Z 
2022-12-01T11:15:25.9057806Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9057915Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9057935Z 
2022-12-01T11:15:25.9058039Z OK (skipped=1)
2022-12-01T11:15:25.9058057Z 
2022-12-01T11:15:25.9058176Z Generating XML reports...
2022-12-01T11:15:25.9058609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111303.xml
2022-12-01T11:15:25.9058974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9059142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9059518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9059713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9059733Z 
2022-12-01T11:15:25.9059838Z Running tests...
2022-12-01T11:15:25.9060098Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9060409Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9060661Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9060863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 119905
2022-12-01T11:15:25.9061076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 119906
2022-12-01T11:15:25.9061442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9061614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9062051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9062236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9062601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9062773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9063135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9063317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9063560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9063799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9064192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9064593Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9064821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9065046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9065283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.9065504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.9065903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.9066294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.9066676Z STAGE:2022-12-01 11:13:09 119905:119905 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9067010Z STAGE:2022-12-01 11:13:09 119906:119906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9067342Z STAGE:2022-12-01 11:13:09 119906:119906 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9067678Z STAGE:2022-12-01 11:13:09 119905:119905 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9068211Z STAGE:2022-12-01 11:13:09 119905:119905 ActivityProfilerController.cpp:294] Completed Stage: Warm UpSTAGE:2022-12-01 11:13:09 119906:119906 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9068232Z 
2022-12-01T11:15:25.9068570Z STAGE:2022-12-01 11:13:09 119905:119905 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9068952Z STAGE:2022-12-01 11:13:09 119906:119906 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9069040Z ok (3.720s)
2022-12-01T11:15:25.9069063Z 
2022-12-01T11:15:25.9069327Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9069659Z Ran 1 test in 3.720s
2022-12-01T11:15:25.9069682Z 
2022-12-01T11:15:25.9069772Z OK
2022-12-01T11:15:25.9069792Z 
2022-12-01T11:15:25.9069916Z Generating XML reports...
2022-12-01T11:15:25.9070365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111305.xml
2022-12-01T11:15:25.9070736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9070910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9071287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9071461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9071557Z 
2022-12-01T11:15:25.9071675Z Running tests...
2022-12-01T11:15:25.9071942Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9072253Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9072502Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9072722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120024
2022-12-01T11:15:25.9072938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120025
2022-12-01T11:15:25.9073306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9073464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9073844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9074037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9074402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9074575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9074951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9075138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9075380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9075614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9076000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9076459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9076698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9076924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9077076Z skip: Skipped due to small world size. (3.604s)
2022-12-01T11:15:25.9077097Z 
2022-12-01T11:15:25.9077364Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9077475Z Ran 1 test in 3.604s
2022-12-01T11:15:25.9077494Z 
2022-12-01T11:15:25.9077599Z OK (skipped=1)
2022-12-01T11:15:25.9077618Z 
2022-12-01T11:15:25.9077723Z Generating XML reports...
2022-12-01T11:15:25.9078169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111311.xml
2022-12-01T11:15:25.9078545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9078718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9079095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9079281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9079301Z 
2022-12-01T11:15:25.9079408Z Running tests...
2022-12-01T11:15:25.9079663Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9079976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9080217Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9080436Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120133
2022-12-01T11:15:25.9080709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120134
2022-12-01T11:15:25.9081088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9081259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9081630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9081818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9082181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9082338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9082713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9082899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9083149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9083393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9083792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9084190Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9084414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9084638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9084723Z ok (3.775s)
2022-12-01T11:15:25.9084743Z 
2022-12-01T11:15:25.9085007Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9085123Z Ran 1 test in 3.776s
2022-12-01T11:15:25.9085143Z 
2022-12-01T11:15:25.9085282Z OK
2022-12-01T11:15:25.9085304Z 
2022-12-01T11:15:25.9085431Z Generating XML reports...
2022-12-01T11:15:25.9085880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111317.xml
2022-12-01T11:15:25.9086245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9086418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9086794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9086966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9086985Z 
2022-12-01T11:15:25.9087091Z Running tests...
2022-12-01T11:15:25.9087349Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9087666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9087905Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9088122Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120242
2022-12-01T11:15:25.9088341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120243
2022-12-01T11:15:25.9088710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9088868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9089249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9089435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9089802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9090027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9090405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9090593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9090834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9091075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9091456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9091851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9092076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9092308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9092405Z ok (3.704s)
2022-12-01T11:15:25.9092425Z 
2022-12-01T11:15:25.9092683Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9092791Z Ran 1 test in 3.704s
2022-12-01T11:15:25.9092811Z 
2022-12-01T11:15:25.9092901Z OK
2022-12-01T11:15:25.9092920Z 
2022-12-01T11:15:25.9093025Z Generating XML reports...
2022-12-01T11:15:25.9093470Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111323.xml
2022-12-01T11:15:25.9093840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9094013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9094388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9094625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9094646Z 
2022-12-01T11:15:25.9094750Z Running tests...
2022-12-01T11:15:25.9095017Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9095327Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9095587Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9095804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120351
2022-12-01T11:15:25.9096003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120352
2022-12-01T11:15:25.9096369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9096538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9096923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9097112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9097477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9097645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9098020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9098205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9098432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9098672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9099165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9099559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9099785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9100012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9100111Z ok (3.665s)
2022-12-01T11:15:25.9100131Z 
2022-12-01T11:15:25.9100391Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9100487Z Ran 1 test in 3.665s
2022-12-01T11:15:25.9100524Z 
2022-12-01T11:15:25.9100597Z OK
2022-12-01T11:15:25.9100617Z 
2022-12-01T11:15:25.9100734Z Generating XML reports...
2022-12-01T11:15:25.9101188Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111328.xml
2022-12-01T11:15:25.9101564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9101739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9102115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9102302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9102322Z 
2022-12-01T11:15:25.9102430Z Running tests...
2022-12-01T11:15:25.9102677Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9102992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9103278Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9103493Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120460
2022-12-01T11:15:25.9103758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120461
2022-12-01T11:15:25.9104135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9104309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9104687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9104874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9105225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9105393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9105768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9105962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9106205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9106450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9106847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9107239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9107451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9107790Z STAGE:2022-12-01 11:13:37 120461:120461 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9108018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9108413Z STAGE:2022-12-01 11:13:38 120460:120460 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9108750Z STAGE:2022-12-01 11:13:38 120460:120460 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9109084Z STAGE:2022-12-01 11:13:38 120461:120461 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9109185Z ok (3.793s)
2022-12-01T11:15:25.9109205Z 
2022-12-01T11:15:25.9109702Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9109821Z Ran 1 test in 3.793s
2022-12-01T11:15:25.9109842Z 
2022-12-01T11:15:25.9109917Z OK
2022-12-01T11:15:25.9109936Z 
2022-12-01T11:15:25.9110060Z Generating XML reports...
2022-12-01T11:15:25.9110517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111334.xml
2022-12-01T11:15:25.9110887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9111067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9111446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9111632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9111651Z 
2022-12-01T11:15:25.9111759Z Running tests...
2022-12-01T11:15:25.9112022Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9112318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9112597Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9112819Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120573
2022-12-01T11:15:25.9113037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120574
2022-12-01T11:15:25.9113481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9113666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9114049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9114240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9114591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9114767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9115145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9115333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9115580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9115819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9116218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9116608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9116835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9117160Z STAGE:2022-12-01 11:13:43 120574:120574 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9117387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9117718Z STAGE:2022-12-01 11:13:43 120573:120573 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9118128Z STAGE:2022-12-01 11:13:43 120573:120573 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9118463Z STAGE:2022-12-01 11:13:43 120574:120574 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9118561Z ok (3.791s)
2022-12-01T11:15:25.9118582Z 
2022-12-01T11:15:25.9118839Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9118949Z Ran 1 test in 3.791s
2022-12-01T11:15:25.9118968Z 
2022-12-01T11:15:25.9119041Z OK
2022-12-01T11:15:25.9119077Z 
2022-12-01T11:15:25.9119183Z Generating XML reports...
2022-12-01T11:15:25.9119634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111340.xml
2022-12-01T11:15:25.9120003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9120179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9120558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9120745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9120764Z 
2022-12-01T11:15:25.9120872Z Running tests...
2022-12-01T11:15:25.9121133Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9121430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9121701Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9121917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120686
2022-12-01T11:15:25.9122132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120687
2022-12-01T11:15:25.9122606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9122789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9123166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9123348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9123709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9123864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9124238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9124421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9124664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9124914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9125310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9125707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9125932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9126143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9126482Z STAGE:2022-12-01 11:13:49 120686:120686 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9126805Z STAGE:2022-12-01 11:13:49 120687:120687 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9127137Z STAGE:2022-12-01 11:13:49 120687:120687 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9127535Z STAGE:2022-12-01 11:13:49 120686:120686 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9127632Z ok (3.712s)
2022-12-01T11:15:25.9127652Z 
2022-12-01T11:15:25.9127910Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9128022Z Ran 1 test in 3.712s
2022-12-01T11:15:25.9128042Z 
2022-12-01T11:15:25.9128131Z OK
2022-12-01T11:15:25.9128150Z 
2022-12-01T11:15:25.9128256Z Generating XML reports...
2022-12-01T11:15:25.9128703Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111346.xml
2022-12-01T11:15:25.9129070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9129239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9129624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9129812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9129832Z 
2022-12-01T11:15:25.9129940Z Running tests...
2022-12-01T11:15:25.9130200Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9130497Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9130735Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T11:15:25.9130755Z 
2022-12-01T11:15:25.9131018Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9131129Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9131148Z 
2022-12-01T11:15:25.9131253Z OK (skipped=1)
2022-12-01T11:15:25.9131272Z 
2022-12-01T11:15:25.9131395Z Generating XML reports...
2022-12-01T11:15:25.9131844Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111351.xml
2022-12-01T11:15:25.9132267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9132445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9132810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9132998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9133017Z 
2022-12-01T11:15:25.9133125Z Running tests...
2022-12-01T11:15:25.9133386Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9133697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9133959Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T11:15:25.9133983Z 
2022-12-01T11:15:25.9134246Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9134356Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9134375Z 
2022-12-01T11:15:25.9134482Z OK (skipped=1)
2022-12-01T11:15:25.9134502Z 
2022-12-01T11:15:25.9134609Z Generating XML reports...
2022-12-01T11:15:25.9135057Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111353.xml
2022-12-01T11:15:25.9135425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9135595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9135974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9136164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9136226Z 
2022-12-01T11:15:25.9136338Z Running tests...
2022-12-01T11:15:25.9136604Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9136899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9137156Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2022-12-01T11:15:25.9137176Z 
2022-12-01T11:15:25.9137437Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9137543Z Ran 1 test in 0.002s
2022-12-01T11:15:25.9137562Z 
2022-12-01T11:15:25.9137664Z OK (skipped=1)
2022-12-01T11:15:25.9137682Z 
2022-12-01T11:15:25.9137803Z Generating XML reports...
2022-12-01T11:15:25.9138242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111355.xml
2022-12-01T11:15:25.9138608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9138785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9139145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9139333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9139353Z 
2022-12-01T11:15:25.9139462Z Running tests...
2022-12-01T11:15:25.9139722Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9140030Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9140291Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9140509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 120898
2022-12-01T11:15:25.9140726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 120899
2022-12-01T11:15:25.9141145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9141312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9141693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9141881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9142244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9142414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9142792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9142979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9143221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9143455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9143854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9144250Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9144476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9144807Z STAGE:2022-12-01 11:14:01 120899:120899 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9145030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9145361Z STAGE:2022-12-01 11:14:01 120898:120898 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9145761Z STAGE:2022-12-01 11:14:01 120899:120899 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9146098Z STAGE:2022-12-01 11:14:01 120898:120898 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9146181Z ok (3.812s)
2022-12-01T11:15:25.9146215Z 
2022-12-01T11:15:25.9146463Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9146576Z Ran 1 test in 3.813s
2022-12-01T11:15:25.9146595Z 
2022-12-01T11:15:25.9146683Z OK
2022-12-01T11:15:25.9146702Z 
2022-12-01T11:15:25.9146826Z Generating XML reports...
2022-12-01T11:15:25.9147271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111357.xml
2022-12-01T11:15:25.9147642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9147816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9148204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9148378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9148397Z 
2022-12-01T11:15:25.9148504Z Running tests...
2022-12-01T11:15:25.9148764Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9149075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9149333Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9149780Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121011
2022-12-01T11:15:25.9150000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121012
2022-12-01T11:15:25.9150377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9150538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9150981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9151179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9151545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9151715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9152090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9152276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9152518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9152762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9153156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9153542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9153769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9153992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9154088Z ok (3.704s)
2022-12-01T11:15:25.9154109Z 
2022-12-01T11:15:25.9154371Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9154482Z Ran 1 test in 3.704s
2022-12-01T11:15:25.9154502Z 
2022-12-01T11:15:25.9154593Z OK
2022-12-01T11:15:25.9154612Z 
2022-12-01T11:15:25.9154717Z Generating XML reports...
2022-12-01T11:15:25.9155241Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111403.xml
2022-12-01T11:15:25.9155612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9155784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9156160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9156347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9156367Z 
2022-12-01T11:15:25.9156473Z Running tests...
2022-12-01T11:15:25.9156734Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9157047Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9157315Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9157544Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121120
2022-12-01T11:15:25.9157761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121121
2022-12-01T11:15:25.9158126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9158298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9158674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9158860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9159225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9159395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9159797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9159997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9160238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9160482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9160879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9161275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9161506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9161733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9162063Z STAGE:2022-12-01 11:14:12 121121:121121 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9162386Z STAGE:2022-12-01 11:14:12 121120:121120 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9162725Z STAGE:2022-12-01 11:14:12 121120:121120 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9163064Z STAGE:2022-12-01 11:14:12 121121:121121 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9163164Z ok (3.705s)
2022-12-01T11:15:25.9163184Z 
2022-12-01T11:15:25.9163445Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9163551Z Ran 1 test in 3.705s
2022-12-01T11:15:25.9163571Z 
2022-12-01T11:15:25.9163660Z OK
2022-12-01T11:15:25.9163679Z 
2022-12-01T11:15:25.9163801Z Generating XML reports...
2022-12-01T11:15:25.9164236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111409.xml
2022-12-01T11:15:25.9164718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9164889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9165263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9165449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9165469Z 
2022-12-01T11:15:25.9165576Z Running tests...
2022-12-01T11:15:25.9165843Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9166158Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9166420Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9166638Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121233
2022-12-01T11:15:25.9166853Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121234
2022-12-01T11:15:25.9167217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9167385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9167758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9167949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9168320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9168491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9168888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9169081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9169367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9169614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9170015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9170407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9170633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9170969Z STAGE:2022-12-01 11:14:18 121234:121234 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9171196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9171522Z STAGE:2022-12-01 11:14:18 121233:121233 ActivityProfilerController.cpp:294] Completed Stage: Warm Up
2022-12-01T11:15:25.9171861Z STAGE:2022-12-01 11:14:18 121233:121233 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9172194Z STAGE:2022-12-01 11:14:18 121234:121234 ActivityProfilerController.cpp:300] Completed Stage: Collection
2022-12-01T11:15:25.9172294Z ok (3.807s)
2022-12-01T11:15:25.9172315Z 
2022-12-01T11:15:25.9172576Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9172687Z Ran 1 test in 3.807s
2022-12-01T11:15:25.9172707Z 
2022-12-01T11:15:25.9172792Z OK
2022-12-01T11:15:25.9172810Z 
2022-12-01T11:15:25.9172928Z Generating XML reports...
2022-12-01T11:15:25.9173365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111415.xml
2022-12-01T11:15:25.9173735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9173963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9174343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9174532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9174552Z 
2022-12-01T11:15:25.9174658Z Running tests...
2022-12-01T11:15:25.9174919Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9175232Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9175499Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9175700Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121346
2022-12-01T11:15:25.9175916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121347
2022-12-01T11:15:25.9176291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9176465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9176843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9177032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9177400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9177574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9177936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9178123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9178372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9178655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9179063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9179453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9179678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9179902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9180003Z ok (3.796s)
2022-12-01T11:15:25.9180022Z 
2022-12-01T11:15:25.9180270Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9180381Z Ran 1 test in 3.797s
2022-12-01T11:15:25.9180405Z 
2022-12-01T11:15:25.9180490Z OK
2022-12-01T11:15:25.9180509Z 
2022-12-01T11:15:25.9180629Z Generating XML reports...
2022-12-01T11:15:25.9181081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111420.xml
2022-12-01T11:15:25.9181450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9181625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9181998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9182183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9182203Z 
2022-12-01T11:15:25.9182294Z Running tests...
2022-12-01T11:15:25.9182561Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9182875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9183202Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9183419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121545
2022-12-01T11:15:25.9183635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121546
2022-12-01T11:15:25.9184000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9184175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9184536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9184727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9185093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9185267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9185651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9185840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9186081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9186315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9186709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9187092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9187320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9187592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9187697Z ok (5.194s)
2022-12-01T11:15:25.9187716Z 
2022-12-01T11:15:25.9187980Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9188085Z Ran 1 test in 5.194s
2022-12-01T11:15:25.9188105Z 
2022-12-01T11:15:25.9188195Z OK
2022-12-01T11:15:25.9188214Z 
2022-12-01T11:15:25.9188332Z Generating XML reports...
2022-12-01T11:15:25.9188765Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111426.xml
2022-12-01T11:15:25.9189135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9189305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9189938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9190134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9190155Z 
2022-12-01T11:15:25.9190264Z Running tests...
2022-12-01T11:15:25.9190527Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9190842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9191102Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9191303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121746
2022-12-01T11:15:25.9191517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121747
2022-12-01T11:15:25.9191887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9192062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9192530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9192714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9193076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9193246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9193622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9193796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9194033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9194277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9194678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9195076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9195301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9195525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9195630Z ok (5.712s)
2022-12-01T11:15:25.9195651Z 
2022-12-01T11:15:25.9195897Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9196012Z Ran 1 test in 5.713s
2022-12-01T11:15:25.9196031Z 
2022-12-01T11:15:25.9196121Z OK
2022-12-01T11:15:25.9196140Z 
2022-12-01T11:15:25.9196262Z Generating XML reports...
2022-12-01T11:15:25.9196716Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111434.xml
2022-12-01T11:15:25.9197146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9197323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9197700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9197889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9197908Z 
2022-12-01T11:15:25.9197998Z Running tests...
2022-12-01T11:15:25.9198258Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9198568Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9198826Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9199042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121861
2022-12-01T11:15:25.9199261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121862
2022-12-01T11:15:25.9199632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9199809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9200172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9200361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9200728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9200897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9201266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9201502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9201748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9201986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9202383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9202760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9202987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9203212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9203464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y89q8jk
2022-12-01T11:15:25.9203722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpspk_wo18
2022-12-01T11:15:25.9203988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y89q8jk/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9204246Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpspk_wo18/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9204344Z ok (3.719s)
2022-12-01T11:15:25.9204364Z 
2022-12-01T11:15:25.9204631Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9204727Z Ran 1 test in 3.719s
2022-12-01T11:15:25.9204747Z 
2022-12-01T11:15:25.9204838Z OK
2022-12-01T11:15:25.9204858Z 
2022-12-01T11:15:25.9204979Z Generating XML reports...
2022-12-01T11:15:25.9205432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111441.xml
2022-12-01T11:15:25.9205798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9205974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9206392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9206586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9206607Z 
2022-12-01T11:15:25.9206697Z Running tests...
2022-12-01T11:15:25.9206958Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9207264Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9207512Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9207732Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 121974
2022-12-01T11:15:25.9207949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 121975
2022-12-01T11:15:25.9208322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9208498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9208873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9209047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9209409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9209578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9209948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9210132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9210373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9210668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9211066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9211444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9211673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9211898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9212152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvv8a8n7e
2022-12-01T11:15:25.9212416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvv8a8n7e/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9212666Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe44fb3un
2022-12-01T11:15:25.9212930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe44fb3un/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9213025Z ok (5.194s)
2022-12-01T11:15:25.9213046Z 
2022-12-01T11:15:25.9213313Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9213409Z Ran 1 test in 5.194s
2022-12-01T11:15:25.9213428Z 
2022-12-01T11:15:25.9213518Z OK
2022-12-01T11:15:25.9213537Z 
2022-12-01T11:15:25.9213656Z Generating XML reports...
2022-12-01T11:15:25.9214104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111447.xml
2022-12-01T11:15:25.9214468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9214639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9215015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9215253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9215274Z 
2022-12-01T11:15:25.9215382Z Running tests...
2022-12-01T11:15:25.9215628Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9215937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9216222Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9216440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122085
2022-12-01T11:15:25.9216650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122086
2022-12-01T11:15:25.9217019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9217192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9217572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9217744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9218107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9218277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9218653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9218842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9219081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9219320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9219770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9220165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9220377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9220600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9220850Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcyjpz0jc
2022-12-01T11:15:25.9221114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcyjpz0jc/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9221358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbofvawyk
2022-12-01T11:15:25.9221616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbofvawyk/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9222396Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.9223212Z [W reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2022-12-01T11:15:25.9223317Z ok (5.787s)
2022-12-01T11:15:25.9223338Z 
2022-12-01T11:15:25.9223604Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9223710Z Ran 1 test in 5.787s
2022-12-01T11:15:25.9223729Z 
2022-12-01T11:15:25.9223817Z OK
2022-12-01T11:15:25.9223836Z 
2022-12-01T11:15:25.9223942Z Generating XML reports...
2022-12-01T11:15:25.9224403Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111454.xml
2022-12-01T11:15:25.9224775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9224946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9225322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9225518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9225538Z 
2022-12-01T11:15:25.9225642Z Running tests...
2022-12-01T11:15:25.9225905Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9226201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9226484Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9226702Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122200
2022-12-01T11:15:25.9226920Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122201
2022-12-01T11:15:25.9227289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9227464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9227948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9228137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9228503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9228657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9229034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9229220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9229688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9229938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9230353Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9230743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9230975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9231194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9231416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.9231650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.9232035Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.9232418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.9232733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.9232979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.9233366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.9233746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.9234004Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8rhzimw
2022-12-01T11:15:25.9234257Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8rhzimw/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9234505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirl7_aq3
2022-12-01T11:15:25.9234773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirl7_aq3/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9234871Z ok (10.205s)
2022-12-01T11:15:25.9234893Z 
2022-12-01T11:15:25.9235158Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9235271Z Ran 1 test in 10.205s
2022-12-01T11:15:25.9235291Z 
2022-12-01T11:15:25.9235381Z OK
2022-12-01T11:15:25.9235400Z 
2022-12-01T11:15:25.9235520Z Generating XML reports...
2022-12-01T11:15:25.9235956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111502.xml
2022-12-01T11:15:25.9236327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9236504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9236882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9237140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9237161Z 
2022-12-01T11:15:25.9237264Z Running tests...
2022-12-01T11:15:25.9237529Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9237838Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2022-12-01T11:15:25.9238125Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2022-12-01T11:15:25.9238329Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 122323
2022-12-01T11:15:25.9238546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 122324
2022-12-01T11:15:25.9238915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9239085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9239467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9239652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9240012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 62 slow tests
2022-12-01T11:15:25.9240187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2022-12-01T11:15:25.9240558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 421 disabled tests
2022-12-01T11:15:25.9240730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2022-12-01T11:15:25.9240971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2022-12-01T11:15:25.9241213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2022-12-01T11:15:25.9241664Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9242072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2022-12-01T11:15:25.9242300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2022-12-01T11:15:25.9242523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2022-12-01T11:15:25.9242761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2022-12-01T11:15:25.9242998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2022-12-01T11:15:25.9243376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.9243762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2022-12-01T11:15:25.9244008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2022-12-01T11:15:25.9244246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2022-12-01T11:15:25.9244630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.9245017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2022-12-01T11:15:25.9245276Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie20eko7
2022-12-01T11:15:25.9245539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie20eko7/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9245783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu00fexmf
2022-12-01T11:15:25.9246091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu00fexmf/_remote_module_non_scriptable.py
2022-12-01T11:15:25.9246191Z ok (10.198s)
2022-12-01T11:15:25.9246211Z 
2022-12-01T11:15:25.9246479Z ----------------------------------------------------------------------
2022-12-01T11:15:25.9246591Z Ran 1 test in 10.198s
2022-12-01T11:15:25.9246612Z 
2022-12-01T11:15:25.9246701Z OK
2022-12-01T11:15:25.9246720Z 
2022-12-01T11:15:25.9246840Z Generating XML reports...
2022-12-01T11:15:25.9247291Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111514.xml
2022-12-01T11:15:25.9247310Z 
2022-12-01T11:15:25.9247722Z ##[endgroup]
2022-12-01T11:15:25.9248190Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_7nnidt1g)
2022-12-01T11:15:25.9248210Z 
2022-12-01T11:15:25.9248465Z Running distributed/_shard/test_sharder ... [2022-12-01 11:15:25.663839]
2022-12-01T11:15:25.9248937Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2022-12-01 11:15:25.664154]
2022-12-01T11:15:27.3502408Z 
2022-12-01T11:15:27.3502878Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2022-12-01T11:15:27.3503812Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_0bdqlt9c)
2022-12-01T11:15:27.3504444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1jsdy7x1
2022-12-01T11:15:27.3504991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1jsdy7x1/_remote_module_non_scriptable.py
2022-12-01T11:15:27.3505297Z 
2022-12-01T11:15:27.3516587Z ##[endgroup]
2022-12-01T11:15:27.3517390Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_0bdqlt9c)
2022-12-01T11:15:27.3517766Z 
2022-12-01T11:15:27.3518285Z Running distributed/pipeline/sync/skip/test_api ... [2022-12-01 11:15:27.350260]
2022-12-01T11:15:27.3518940Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2022-12-01 11:15:27.350520]
2022-12-01T11:15:29.7591203Z 
2022-12-01T11:15:29.7591694Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api
2022-12-01T11:15:29.7592752Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_r3_urmwq)
2022-12-01T11:15:29.7593304Z ============================= test session starts ==============================
2022-12-01T11:15:29.7593922Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:29.7594260Z cachedir: .pytest_cache
2022-12-01T11:15:29.7594838Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:29.7595310Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:29.7595625Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:29.7596195Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:29.7596590Z collecting ... collected 3 items
2022-12-01T11:15:29.7597201Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr
2022-12-01T11:15:29.7597658Z 
2022-12-01T11:15:29.7597873Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2022-12-01T11:15:29.7598345Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2022-12-01T11:15:29.7598799Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2022-12-01T11:15:29.7599290Z 
2022-12-01T11:15:29.7599457Z ============================== 3 passed in 0.04s ===============================
2022-12-01T11:15:29.7599655Z 
2022-12-01T11:15:29.7599942Z ##[endgroup]
2022-12-01T11:15:29.7600596Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_r3_urmwq)
2022-12-01T11:15:29.7600981Z 
2022-12-01T11:15:29.7601268Z Running distributed/pipeline/sync/skip/test_leak ... [2022-12-01 11:15:29.759167]
2022-12-01T11:15:29.7601882Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2022-12-01 11:15:29.759431]
2022-12-01T11:15:31.9412045Z 
2022-12-01T11:15:31.9412716Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak
2022-12-01T11:15:31.9413788Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_814721jz)
2022-12-01T11:15:31.9414638Z ============================= test session starts ==============================
2022-12-01T11:15:31.9415258Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:31.9415615Z cachedir: .pytest_cache
2022-12-01T11:15:31.9416194Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:31.9416879Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:31.9417277Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:31.9417853Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:31.9418243Z collecting ... collected 8 items
2022-12-01T11:15:31.9420543Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval]
2022-12-01T11:15:31.9421927Z 
2022-12-01T11:15:31.9422614Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
2022-12-01T11:15:31.9423567Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
2022-12-01T11:15:31.9424553Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
2022-12-01T11:15:31.9425521Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
2022-12-01T11:15:31.9426511Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
2022-12-01T11:15:31.9427389Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
2022-12-01T11:15:31.9428041Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
2022-12-01T11:15:31.9428768Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]
2022-12-01T11:15:31.9429253Z 
2022-12-01T11:15:31.9430052Z ============================== 8 passed in 0.27s ===============================
2022-12-01T11:15:31.9430341Z 
2022-12-01T11:15:31.9430680Z ##[endgroup]
2022-12-01T11:15:31.9431375Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_814721jz)
2022-12-01T11:15:31.9431942Z 
2022-12-01T11:15:31.9432255Z Running distributed/pipeline/sync/skip/test_tracker ... [2022-12-01 11:15:31.941264]
2022-12-01T11:15:31.9432869Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2022-12-01 11:15:31.941568]
2022-12-01T11:15:35.3019066Z 
2022-12-01T11:15:35.3019607Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker
2022-12-01T11:15:35.3021029Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_5xtceief)
2022-12-01T11:15:35.3021829Z ============================= test session starts ==============================
2022-12-01T11:15:35.3022449Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:35.3022801Z cachedir: .pytest_cache
2022-12-01T11:15:35.3023356Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:35.3023945Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:35.3024449Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:35.3025005Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:35.3025399Z collecting ... collected 6 items
2022-12-01T11:15:35.3026389Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing
2022-12-01T11:15:35.3027208Z 
2022-12-01T11:15:35.3027668Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2022-12-01T11:15:35.3028195Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%]
2022-12-01T11:15:35.3028677Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2022-12-01T11:15:35.3029110Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2022-12-01T11:15:35.3030197Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2022-12-01T11:15:35.3031134Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2022-12-01T11:15:35.3031643Z 
2022-12-01T11:15:35.3031812Z ============================== 6 passed in 1.37s ===============================
2022-12-01T11:15:35.3032009Z 
2022-12-01T11:15:35.3032313Z ##[endgroup]
2022-12-01T11:15:35.3033012Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_5xtceief)
2022-12-01T11:15:35.3033422Z 
2022-12-01T11:15:35.3033703Z Running distributed/pipeline/sync/test_bugs ... [2022-12-01 11:15:35.301866]
2022-12-01T11:15:35.3034283Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2022-12-01 11:15:35.302163]
2022-12-01T11:15:41.0751972Z 
2022-12-01T11:15:41.0752740Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs
2022-12-01T11:15:41.0754304Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_o2iq64bs)
2022-12-01T11:15:41.0755140Z ============================= test session starts ==============================
2022-12-01T11:15:41.0755780Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:41.0756116Z cachedir: .pytest_cache
2022-12-01T11:15:41.0757108Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:41.0757778Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:41.0758094Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:41.0758658Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:41.0759047Z collecting ... collected 4 items
2022-12-01T11:15:41.0759722Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms
2022-12-01T11:15:41.0760250Z 
2022-12-01T11:15:41.0760478Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
2022-12-01T11:15:41.0760911Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
2022-12-01T11:15:41.0761354Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait PASSED           [ 75%]
2022-12-01T11:15:41.0761776Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]
2022-12-01T11:15:41.0762024Z 
2022-12-01T11:15:41.0762166Z ============================== 4 passed in 3.74s ===============================
2022-12-01T11:15:41.0762363Z 
2022-12-01T11:15:41.0762684Z ##[endgroup]
2022-12-01T11:15:41.0763310Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_o2iq64bs)
2022-12-01T11:15:41.0763677Z 
2022-12-01T11:15:41.0763987Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2022-12-01 11:15:41.075194]
2022-12-01T11:15:41.0764625Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2022-12-01 11:15:41.075492]
2022-12-01T11:15:43.5791409Z 
2022-12-01T11:15:43.5792210Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm
2022-12-01T11:15:43.5793295Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_1jfgv8ko)
2022-12-01T11:15:43.5793865Z ============================= test session starts ==============================
2022-12-01T11:15:43.5794475Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:43.5794836Z cachedir: .pytest_cache
2022-12-01T11:15:43.5795396Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:43.5795842Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:43.5796173Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:43.5796713Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:43.5797122Z collecting ... collected 11 items
2022-12-01T11:15:43.5799264Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad
2022-12-01T11:15:43.5800657Z 
2022-12-01T11:15:43.5800994Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2022-12-01T11:15:43.5801577Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2022-12-01T11:15:43.5802163Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2022-12-01T11:15:43.5802743Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2022-12-01T11:15:43.5803211Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2022-12-01T11:15:43.5803688Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2022-12-01T11:15:43.5804199Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2022-12-01T11:15:43.5804670Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2022-12-01T11:15:43.5805110Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2022-12-01T11:15:43.5805564Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2022-12-01T11:15:43.5806030Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2022-12-01T11:15:43.5806299Z 
2022-12-01T11:15:43.5806443Z ============================== 11 passed in 0.58s ==============================
2022-12-01T11:15:43.5806640Z 
2022-12-01T11:15:43.5806949Z ##[endgroup]
2022-12-01T11:15:43.5807641Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_1jfgv8ko)
2022-12-01T11:15:43.5808046Z 
2022-12-01T11:15:43.5808341Z Running distributed/pipeline/sync/test_microbatch ... [2022-12-01 11:15:43.579099]
2022-12-01T11:15:43.5809010Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2022-12-01 11:15:43.579398]
2022-12-01T11:15:45.5435835Z 
2022-12-01T11:15:45.5436549Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch
2022-12-01T11:15:45.5437522Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_ulcva8cy)
2022-12-01T11:15:45.5438089Z ============================= test session starts ==============================
2022-12-01T11:15:45.5438704Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:45.5439061Z cachedir: .pytest_cache
2022-12-01T11:15:45.5439618Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:45.5440054Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:45.5440384Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:45.5440955Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:45.5441352Z collecting ... collected 10 items
2022-12-01T11:15:45.5442603Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors
2022-12-01T11:15:45.5443917Z 
2022-12-01T11:15:45.5444145Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
2022-12-01T11:15:45.5444599Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
2022-12-01T11:15:45.5445026Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
2022-12-01T11:15:45.5445499Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
2022-12-01T11:15:45.5445965Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
2022-12-01T11:15:45.5446411Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
2022-12-01T11:15:45.5446820Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
2022-12-01T11:15:45.5447260Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
2022-12-01T11:15:45.5447692Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
2022-12-01T11:15:45.5448165Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%]
2022-12-01T11:15:45.5448413Z 
2022-12-01T11:15:45.5448573Z ============================== 10 passed in 0.06s ==============================
2022-12-01T11:15:45.5448765Z 
2022-12-01T11:15:45.5449071Z ##[endgroup]
2022-12-01T11:15:45.5449731Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_ulcva8cy)
2022-12-01T11:15:45.5450124Z 
2022-12-01T11:15:45.5450395Z Running distributed/pipeline/sync/test_pipeline ... [2022-12-01 11:15:45.543614]
2022-12-01T11:15:45.5451013Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2022-12-01 11:15:45.543903]
2022-12-01T11:15:47.4842136Z 
2022-12-01T11:15:47.4842677Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline
2022-12-01T11:15:47.4844025Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_q8et8wiw)
2022-12-01T11:15:47.4844809Z ============================= test session starts ==============================
2022-12-01T11:15:47.4845658Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:47.4846024Z cachedir: .pytest_cache
2022-12-01T11:15:47.4846582Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:47.4847029Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:47.4847399Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:47.4848120Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:47.4848500Z collecting ... collected 1 item
2022-12-01T11:15:47.4848912Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles
2022-12-01T11:15:47.4849195Z 
2022-12-01T11:15:47.4849422Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2022-12-01T11:15:47.4849677Z 
2022-12-01T11:15:47.4849820Z ============================== 1 passed in 0.03s ===============================
2022-12-01T11:15:47.4850032Z 
2022-12-01T11:15:47.4850353Z ##[endgroup]
2022-12-01T11:15:47.4851000Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_q8et8wiw)
2022-12-01T11:15:47.4851384Z 
2022-12-01T11:15:47.4851667Z Running distributed/pipeline/sync/test_worker ... [2022-12-01 11:15:47.484270]
2022-12-01T11:15:47.4852256Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2022-12-01 11:15:47.484572]
2022-12-01T11:15:49.4476344Z 
2022-12-01T11:15:49.4476903Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker
2022-12-01T11:15:49.4478191Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_aqp93wbu)
2022-12-01T11:15:49.4478753Z ============================= test session starts ==============================
2022-12-01T11:15:49.4479358Z platform linux -- Python 3.10.4, pytest-7.1.3, pluggy-1.0.0 -- /opt/conda/bin/python
2022-12-01T11:15:49.4479719Z cachedir: .pytest_cache
2022-12-01T11:15:49.4480291Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2022-12-01T11:15:49.4480714Z torch: 1.13.0a0+gitc13d400
2022-12-01T11:15:49.4481049Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2022-12-01T11:15:49.4481612Z plugins: hypothesis-5.35.1, forked-1.4.0, rerunfailures-10.2, shard-0.1.2, xdist-2.5.0, xdoctest-1.0.2
2022-12-01T11:15:49.4482005Z collecting ... collected 6 items
2022-12-01T11:15:49.4482861Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device
2022-12-01T11:15:49.4483575Z 
2022-12-01T11:15:49.4483810Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%]
2022-12-01T11:15:49.4484263Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 33%]
2022-12-01T11:15:49.4484709Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 50%]
2022-12-01T11:15:49.4485129Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 66%]
2022-12-01T11:15:49.4485560Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 83%]
2022-12-01T11:15:49.4485994Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]
2022-12-01T11:15:49.4486347Z 
2022-12-01T11:15:49.4486525Z ============================== 6 passed in 0.05s ===============================
2022-12-01T11:15:49.4486723Z 
2022-12-01T11:15:49.4487018Z ##[endgroup]
2022-12-01T11:15:49.4487663Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_aqp93wbu)
2022-12-01T11:15:49.4488043Z 
2022-12-01T11:15:49.7145278Z 
2022-12-01T11:15:49.7145638Z real	55m13.824s
2022-12-01T11:15:49.7146044Z user	192m37.972s
2022-12-01T11:15:49.7146467Z sys	122m39.004s
2022-12-01T11:15:49.7146705Z + assert_git_not_dirty
2022-12-01T11:15:49.7147238Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-12-01T11:15:49.7147955Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *xla* ]]
2022-12-01T11:15:49.7148819Z ++ git status --porcelain
2022-12-01T11:15:50.8609545Z + git_status=
2022-12-01T11:15:50.8609993Z + [[ -n '' ]]
2022-12-01T11:15:50.8610407Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 == *cuda* ]]
2022-12-01T11:15:50.8610729Z + [[ 1 == 1 ]]
2022-12-01T11:15:50.8611058Z + echo 'Testing distributed C++ tests'
2022-12-01T11:15:50.8611339Z Testing distributed C++ tests
2022-12-01T11:15:50.8612968Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T11:15:50.8625432Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T11:15:50.8636395Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2022-12-01T11:15:50.8636894Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2022-12-01T11:15:50.8651602Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2022-12-01T11:15:51.1923486Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-12-01T11:15:51.1924099Z [==========] Running 4 tests from 1 test suite.
2022-12-01T11:15:51.1924502Z [----------] Global test environment set-up.
2022-12-01T11:15:51.1924915Z [----------] 4 tests from FileStoreTest
2022-12-01T11:15:51.1925333Z [ RUN      ] FileStoreTest.testGetAndSet
2022-12-01T11:15:51.1928589Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2022-12-01T11:15:51.1929055Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2022-12-01T11:15:51.1932915Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2022-12-01T11:15:51.1933384Z [ RUN      ] FileStoreTest.testStressStore
2022-12-01T11:15:51.2132439Z [       OK ] FileStoreTest.testStressStore (19 ms)
2022-12-01T11:15:51.2132908Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2022-12-01T11:15:51.2333701Z [       OK ] FileStoreTest.testStressStoreWithPrefix (20 ms)
2022-12-01T11:15:51.2334211Z [----------] 4 tests from FileStoreTest (41 ms total)
2022-12-01T11:15:51.2334433Z 
2022-12-01T11:15:51.2334651Z [----------] Global test environment tear-down
2022-12-01T11:15:51.2335663Z [==========] 4 tests from 1 test suite ran. (41 ms total)
2022-12-01T11:15:51.2336039Z [  PASSED  ] 4 tests.
2022-12-01T11:15:51.2986938Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2022-12-01T11:15:51.6302538Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-12-01T11:15:51.6303155Z [==========] Running 4 tests from 1 test suite.
2022-12-01T11:15:51.6303552Z [----------] Global test environment set-up.
2022-12-01T11:15:51.6303960Z [----------] 4 tests from HashStoreTest
2022-12-01T11:15:51.6304355Z [ RUN      ] HashStoreTest.testGetAndSet
2022-12-01T11:15:51.7306981Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2022-12-01T11:15:51.7307470Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2022-12-01T11:15:51.8311160Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2022-12-01T11:15:51.8311657Z [ RUN      ] HashStoreTest.testStressStore
2022-12-01T11:15:51.8319342Z [       OK ] HashStoreTest.testStressStore (0 ms)
2022-12-01T11:15:51.8320326Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2022-12-01T11:15:51.8327174Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2022-12-01T11:15:51.8327716Z [----------] 4 tests from HashStoreTest (202 ms total)
2022-12-01T11:15:51.8327924Z 
2022-12-01T11:15:51.8328166Z [----------] Global test environment tear-down
2022-12-01T11:15:51.8330149Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2022-12-01T11:15:51.8330535Z [  PASSED  ] 4 tests.
2022-12-01T11:15:51.9062828Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2022-12-01T11:15:52.2344593Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-12-01T11:15:52.2345214Z [==========] Running 11 tests from 1 test suite.
2022-12-01T11:15:52.2345914Z [----------] Global test environment set-up.
2022-12-01T11:15:52.2346324Z [----------] 11 tests from TCPStoreTest
2022-12-01T11:15:52.2346713Z [ RUN      ] TCPStoreTest.testHelper
2022-12-01T11:15:53.2503177Z [       OK ] TCPStoreTest.testHelper (1015 ms)
2022-12-01T11:15:53.2504017Z [ RUN      ] TCPStoreTest.testHelperPrefix
2022-12-01T11:15:54.2232019Z [       OK ] TCPStoreTest.testHelperPrefix (972 ms)
2022-12-01T11:15:54.2232509Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2022-12-01T11:15:54.2382066Z [       OK ] TCPStoreTest.testWatchKeyCallback (15 ms)
2022-12-01T11:15:54.2382620Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2022-12-01T11:15:54.2523873Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (14 ms)
2022-12-01T11:15:54.2524389Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2022-12-01T11:15:54.4599574Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (207 ms)
2022-12-01T11:15:54.4600100Z [ RUN      ] TCPStoreTest.testKeyUpdate
2022-12-01T11:15:54.4606770Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2022-12-01T11:15:54.4607219Z [ RUN      ] TCPStoreTest.testKeyCreate
2022-12-01T11:15:54.4612766Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2022-12-01T11:15:54.4613215Z [ RUN      ] TCPStoreTest.testKeyAdd
2022-12-01T11:15:54.4619315Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2022-12-01T11:15:54.4619774Z [ RUN      ] TCPStoreTest.testKeyDelete
2022-12-01T11:15:54.6675110Z [       OK ] TCPStoreTest.testKeyDelete (205 ms)
2022-12-01T11:15:54.6675561Z [ RUN      ] TCPStoreTest.testCleanShutdown
2022-12-01T11:15:54.6682984Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2022-12-01T11:15:54.6683457Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2022-12-01T11:15:54.6696629Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2022-12-01T11:15:54.6697305Z [----------] 11 tests from TCPStoreTest (2435 ms total)
2022-12-01T11:15:54.6697560Z 
2022-12-01T11:15:54.6697806Z [----------] Global test environment tear-down
2022-12-01T11:15:54.6699858Z [==========] 11 tests from 1 test suite ran. (2435 ms total)
2022-12-01T11:15:54.6700216Z [  PASSED  ] 11 tests.
2022-12-01T11:15:54.7412878Z ++ command -v mpiexec
2022-12-01T11:15:54.7415470Z + MPIEXEC=/usr/bin/mpiexec
2022-12-01T11:15:54.7415836Z + [[ -n /usr/bin/mpiexec ]]
2022-12-01T11:15:54.7416108Z + [[ -z true ]]
2022-12-01T11:15:54.7416783Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2022-12-01T11:15:55.1668144Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-12-01T11:15:55.1668767Z [==========] Running 12 tests from 1 test suite.
2022-12-01T11:15:55.1669227Z [----------] Global test environment set-up.
2022-12-01T11:15:55.1670018Z [----------] 12 tests from ProcessGroupGlooTest
2022-12-01T11:15:55.1670505Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2022-12-01T11:15:56.2015572Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1034 ms)
2022-12-01T11:15:56.2016203Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2022-12-01T11:15:56.2367138Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (35 ms)
2022-12-01T11:15:56.2367767Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2022-12-01T11:15:56.5268467Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (290 ms)
2022-12-01T11:15:56.5269032Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2022-12-01T11:15:56.5695510Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2022-12-01T11:15:56.5696050Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2022-12-01T11:15:56.7237233Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (154 ms)
2022-12-01T11:15:56.7237829Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2022-12-01T11:15:56.7655485Z [       OK ] ProcessGroupGlooTest.testBarrier (41 ms)
2022-12-01T11:15:56.7656028Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2022-12-01T11:15:57.7869041Z [E ProcessGroupGloo.cpp:137] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2022-12-01T11:15:57.8073532Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1041 ms)
2022-12-01T11:15:57.8074122Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2022-12-01T11:15:57.9608364Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (153 ms)
2022-12-01T11:15:57.9608873Z [ RUN      ] ProcessGroupGlooTest.testSend
2022-12-01T11:15:58.0029886Z [       OK ] ProcessGroupGlooTest.testSend (42 ms)
2022-12-01T11:15:58.0030584Z [ RUN      ] ProcessGroupGlooTest.testRecv
2022-12-01T11:15:58.0450169Z [       OK ] ProcessGroupGlooTest.testRecv (42 ms)
2022-12-01T11:15:58.0450672Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2022-12-01T11:15:58.0960504Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (51 ms)
2022-12-01T11:15:58.0961016Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2022-12-01T11:15:58.2385231Z [       OK ] ProcessGroupGlooTest.testWaitDelay (142 ms)
2022-12-01T11:15:58.2385783Z [----------] 12 tests from ProcessGroupGlooTest (3071 ms total)
2022-12-01T11:15:58.2386048Z 
2022-12-01T11:15:58.2386306Z [----------] Global test environment tear-down
2022-12-01T11:15:58.2388668Z [==========] 12 tests from 1 test suite ran. (3071 ms total)
2022-12-01T11:15:58.2389082Z [  PASSED  ] 12 tests.
2022-12-01T11:15:58.3401158Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2022-12-01T11:15:58.7798342Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-12-01T11:15:58.7799161Z [==========] Running 11 tests from 1 test suite.
2022-12-01T11:15:58.7799565Z [----------] Global test environment set-up.
2022-12-01T11:15:58.7799997Z [----------] 11 tests from ProcessGroupNCCLTest
2022-12-01T11:15:58.7800447Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2022-12-01T11:16:03.2804995Z [       OK ] ProcessGroupNCCLTest.testAllreduce (4500 ms)
2022-12-01T11:16:03.2805936Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2022-12-01T11:16:06.7431146Z [       OK ] ProcessGroupNCCLTest.testBroadcast (3462 ms)
2022-12-01T11:16:06.7431663Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2022-12-01T11:16:10.2041339Z [       OK ] ProcessGroupNCCLTest.testReduce (3460 ms)
2022-12-01T11:16:10.2041883Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2022-12-01T11:16:11.9728182Z [       OK ] ProcessGroupNCCLTest.testAllgather (1768 ms)
2022-12-01T11:16:11.9728723Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2022-12-01T11:16:13.7252559Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (1752 ms)
2022-12-01T11:16:13.7253112Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2022-12-01T11:16:15.4902494Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (1764 ms)
2022-12-01T11:16:15.4903049Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2022-12-01T11:16:15.5993900Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (109 ms)
2022-12-01T11:16:15.5994590Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2022-12-01T11:16:18.6012864Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3001 ms)
2022-12-01T11:16:18.6013641Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2022-12-01T11:16:21.6027164Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2022-12-01T11:16:21.6027837Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2022-12-01T11:16:23.6986478Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (2095 ms)
2022-12-01T11:16:23.6987047Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2022-12-01T11:16:23.7422713Z [       OK ] ProcessGroupNCCLTest.testBackendName (43 ms)
2022-12-01T11:16:23.7423740Z [----------] 11 tests from ProcessGroupNCCLTest (24962 ms total)
2022-12-01T11:16:23.7424183Z 
2022-12-01T11:16:23.7426979Z [----------] Global test environment tear-down
2022-12-01T11:16:23.7427918Z [==========] 11 tests from 1 test suite ran. (24962 ms total)
2022-12-01T11:16:23.7428305Z [  PASSED  ] 11 tests.
2022-12-01T11:16:23.9539306Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2022-12-01T11:16:24.3897399Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2022-12-01T11:16:24.3898228Z [==========] Running 3 tests from 1 test suite.
2022-12-01T11:16:24.3898794Z [----------] Global test environment set-up.
2022-12-01T11:16:24.3899253Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2022-12-01T11:16:24.3899766Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2022-12-01T11:16:26.8327784Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2442 ms)
2022-12-01T11:16:26.8328495Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2022-12-01T11:16:29.9366954Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3103 ms)
2022-12-01T11:16:29.9367686Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2022-12-01T11:16:30.0401614Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (103 ms)
2022-12-01T11:16:30.0402387Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5650 ms total)
2022-12-01T11:16:30.0402699Z 
2022-12-01T11:16:30.0403141Z [----------] Global test environment tear-down
2022-12-01T11:16:30.0403602Z [==========] 3 tests from 1 test suite ran. (5650 ms total)
2022-12-01T11:16:30.0403938Z [  PASSED  ] 3 tests.
2022-12-01T11:16:30.2538196Z + [[ 1 == 1 ]]
2022-12-01T11:16:30.2538738Z + test_rpc
2022-12-01T11:16:30.2539567Z + [[ linux-bionic-cuda11.6-py3.10-gcc7 != *rocm* ]]
2022-12-01T11:16:30.2540244Z + echo 'Testing RPC C++ tests'
2022-12-01T11:16:30.2540744Z Testing RPC C++ tests
2022-12-01T11:16:30.2544010Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cpp.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_cu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T11:16:30.2560075Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T11:16:30.2571769Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin
2022-12-01T11:16:30.2581860Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2022-12-01T11:16:30.2582602Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2022-12-01T11:16:30.2596369Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2022-12-01T11:16:30.7153095Z [==========] Running 8 tests from 3 test suites.
2022-12-01T11:16:30.7153570Z [----------] Global test environment set-up.
2022-12-01T11:16:30.7154122Z [----------] 4 tests from WireSerialize
2022-12-01T11:16:30.7154509Z [ RUN      ] WireSerialize.Base
2022-12-01T11:16:30.7167550Z [       OK ] WireSerialize.Base (1 ms)
2022-12-01T11:16:30.7269140Z [ RUN      ] WireSerialize.RecopySparseTensors
2022-12-01T11:16:30.7269958Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2022-12-01T11:16:30.7270447Z [ RUN      ] WireSerialize.CloneSparseTensors
2022-12-01T11:16:30.7363504Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2022-12-01T11:16:30.7363942Z [ RUN      ] WireSerialize.Errors
2022-12-01T11:16:30.7391281Z [       OK ] WireSerialize.Errors (2 ms)
2022-12-01T11:16:30.7391742Z [----------] 4 tests from WireSerialize (23 ms total)
2022-12-01T11:16:30.7391968Z 
2022-12-01T11:16:30.7392198Z [----------] 1 test from TestE2ETensorPipe
2022-12-01T11:16:30.7392620Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2022-12-01T11:16:31.4645587Z [W tensorpipe_agent.cpp:726] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2022-12-01T11:16:31.4664461Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (727 ms)
2022-12-01T11:16:31.4665425Z [----------] 1 test from TestE2ETensorPipe (727 ms total)
2022-12-01T11:16:31.4665687Z 
2022-12-01T11:16:31.4665937Z [----------] 3 tests from TensorpipeSerialize
2022-12-01T11:16:31.4666333Z [ RUN      ] TensorpipeSerialize.Base
2022-12-01T11:16:31.4666781Z [       OK ] TensorpipeSerialize.Base (0 ms)
2022-12-01T11:16:31.4667616Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2022-12-01T11:16:31.4766269Z [       OK ] TensorpipeSerialize.RecopySparseTensors (9 ms)
2022-12-01T11:16:31.4767210Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2022-12-01T11:16:31.4768175Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2022-12-01T11:16:31.4769094Z [----------] 3 tests from TensorpipeSerialize (10 ms total)
2022-12-01T11:16:31.4769524Z 
2022-12-01T11:16:31.4769923Z [----------] Global test environment tear-down
2022-12-01T11:16:31.4770773Z [==========] 8 tests from 3 test suites ran. (761 ms total)
2022-12-01T11:16:31.4771414Z [  PASSED  ] 8 tests.
2022-12-01T11:16:31.4771723Z 
2022-12-01T11:16:31.4772029Z   YOU HAVE 1 DISABLED TEST
2022-12-01T11:16:31.4772349Z 
2022-12-01T11:16:31.5819729Z 
2022-12-01T11:16:31.5857097Z Prepare all required actions
2022-12-01T11:16:31.5857557Z Getting action download info
2022-12-01T11:16:31.7980155Z ##[group]Run ./.github/actions/get-workflow-job-id
2022-12-01T11:16:31.7980454Z with:
2022-12-01T11:16:31.7980902Z   github-token: ***
2022-12-01T11:16:31.7981123Z env:
2022-12-01T11:16:31.7981362Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:31.7981632Z   GPU_FLAG: --gpus all
2022-12-01T11:16:31.7981862Z ##[endgroup]
2022-12-01T11:16:31.8016178Z ##[group]Run nick-fields/retry@7d4a37704547a311dbb66ebdf5b23ec19374a767
2022-12-01T11:16:31.8016491Z with:
2022-12-01T11:16:31.8016694Z   shell: bash
2022-12-01T11:16:31.8016938Z   timeout_minutes: 10
2022-12-01T11:16:31.8017191Z   max_attempts: 5
2022-12-01T11:16:31.8017438Z   retry_wait_seconds: 30
2022-12-01T11:16:31.8017978Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "::set-output name=job-id::${GHA_WORKFLOW_JOB_ID}"

2022-12-01T11:16:31.8018609Z   polling_interval_seconds: 1
2022-12-01T11:16:31.8018878Z   warning_on_retry: true
2022-12-01T11:16:31.8019138Z   continue_on_error: false
2022-12-01T11:16:31.8019360Z env:
2022-12-01T11:16:31.8019597Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:31.8019860Z   GPU_FLAG: --gpus all
2022-12-01T11:16:31.8020254Z   GITHUB_TOKEN: ***
2022-12-01T11:16:31.8020482Z ##[endgroup]
2022-12-01T11:16:31.8577340Z 
2022-12-01T11:16:31.8586603Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T11:16:31.8643395Z + python3 -m pip install requests==2.26.0
2022-12-01T11:16:32.1595376Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T11:16:32.1832524Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2022-12-01T11:16:32.2020182Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2022-12-01T11:16:32.2047366Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.9.24)
2022-12-01T11:16:32.2059842Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2022-12-01T11:16:32.2076535Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.13)
2022-12-01T11:16:32.4915517Z ++ python3 .github/scripts/get_workflow_job_id.py 3591403534 i-0d2a7b5e684093f31
2022-12-01T11:16:34.3484294Z + GHA_WORKFLOW_JOB_ID=9818608304
2022-12-01T11:16:34.3485299Z + echo '::set-output name=job-id::9818608304'
2022-12-01T11:16:34.3487569Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T11:16:34.8651148Z Command completed after 1 attempt(s).
2022-12-01T11:16:34.8651783Z 
2022-12-01T11:16:34.8654062Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2022-12-01T11:16:34.8802747Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2022-12-01T11:16:34.8803127Z kill "$MONITOR_SCRIPT_PID"
2022-12-01T11:16:34.8817337Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:34.8817656Z env:
2022-12-01T11:16:34.8817894Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:34.8818174Z   GPU_FLAG: --gpus all
2022-12-01T11:16:34.8818455Z   MONITOR_SCRIPT_PID: 5973
2022-12-01T11:16:34.8818715Z ##[endgroup]
2022-12-01T11:16:34.8916998Z Prepare all required actions
2022-12-01T11:16:34.8917427Z Getting action download info
2022-12-01T11:16:35.1006012Z Download action repository 'actions/upload-artifact@v2' (SHA:82c141cc518b40d92cc801eee768e7aafc9c2fa2)
2022-12-01T11:16:35.2579003Z ##[group]Run ./.github/actions/upload-test-artifacts
2022-12-01T11:16:35.2579313Z with:
2022-12-01T11:16:35.2579674Z   file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304
2022-12-01T11:16:35.2580007Z env:
2022-12-01T11:16:35.2580252Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:35.2580524Z   GPU_FLAG: --gpus all
2022-12-01T11:16:35.2580755Z ##[endgroup]
2022-12-01T11:16:35.2615771Z ##[group]Run # Remove any previous test jsons if they exist
2022-12-01T11:16:35.2616163Z # Remove any previous test jsons if they exist
2022-12-01T11:16:35.2616482Z rm -f test-jsons-*.zip
2022-12-01T11:16:35.2616825Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2022-12-01T11:16:35.2629988Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:35.2630561Z env:
2022-12-01T11:16:35.2630976Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:35.2631277Z   GPU_FLAG: --gpus all
2022-12-01T11:16:35.2631660Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304
2022-12-01T11:16:35.2632006Z ##[endgroup]
2022-12-01T11:16:35.2779274Z   adding: test/allowlist_for_publicAPI.json (deflated 80%)
2022-12-01T11:16:35.2816016Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2022-12-01T11:16:35.2823603Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2022-12-01T11:16:35.2824815Z   adding: test/.pytorch-slow-tests.json (deflated 76%)
2022-12-01T11:16:35.2837234Z   adding: test/.pytorch-disabled-tests.json (deflated 86%)
2022-12-01T11:16:35.2864601Z ##[group]Run # Remove any previous test reports if they exist
2022-12-01T11:16:35.2865005Z # Remove any previous test reports if they exist
2022-12-01T11:16:35.2865355Z rm -f test-reports-*.zip
2022-12-01T11:16:35.2865709Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml'
2022-12-01T11:16:35.2878704Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:35.2879017Z env:
2022-12-01T11:16:35.2879264Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:35.2879517Z   GPU_FLAG: --gpus all
2022-12-01T11:16:35.2879895Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304
2022-12-01T11:16:35.2880256Z ##[endgroup]
2022-12-01T11:16:35.3105321Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20221201102041.xml (deflated 68%)
2022-12-01T11:16:35.3106324Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20221201102041.xml (deflated 43%)
2022-12-01T11:16:35.3107380Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20221201102041.xml (deflated 45%)
2022-12-01T11:16:35.3108509Z   adding: test/test-reports/python-unittest/distributed._shard.checkpoint.test_utils/TEST-TestMedatadaIndex-20221201102103.xml (deflated 72%)
2022-12-01T11:16:35.3109308Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParams-20221201102106.xml (deflated 81%)
2022-12-01T11:16:35.3110582Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDA-20221201102106.xml (deflated 81%)
2022-12-01T11:16:35.3111435Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_flatten_params_wrapper/TEST-TestFlattenParamsCUDAHalf-20221201102106.xml (deflated 81%)
2022-12-01T11:16:35.3112272Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102113.xml (deflated 41%)
2022-12-01T11:16:35.3113251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102122.xml (deflated 42%)
2022-12-01T11:16:35.3114050Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102124.xml (deflated 42%)
2022-12-01T11:16:35.3114842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml (deflated 42%)
2022-12-01T11:16:35.3115633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml (deflated 41%)
2022-12-01T11:16:35.3116402Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102141.xml (deflated 41%)
2022-12-01T11:16:35.3139198Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102147.xml (deflated 41%)
2022-12-01T11:16:35.3140061Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102156.xml (deflated 41%)
2022-12-01T11:16:35.3141064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102204.xml (deflated 41%)
2022-12-01T11:16:35.3141855Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102212.xml (deflated 40%)
2022-12-01T11:16:35.3142633Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102221.xml (deflated 40%)
2022-12-01T11:16:35.3143440Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102229.xml (deflated 40%)
2022-12-01T11:16:35.3144229Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102238.xml (deflated 40%)
2022-12-01T11:16:35.3145030Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102246.xml (deflated 42%)
2022-12-01T11:16:35.3145796Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102250.xml (deflated 42%)
2022-12-01T11:16:35.3146576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102256.xml (deflated 42%)
2022-12-01T11:16:35.3147356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102300.xml (deflated 42%)
2022-12-01T11:16:35.3148137Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102308.xml (deflated 42%)
2022-12-01T11:16:35.3148892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102310.xml (deflated 46%)
2022-12-01T11:16:35.3150241Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102316.xml (deflated 47%)
2022-12-01T11:16:35.3151040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102323.xml (deflated 48%)
2022-12-01T11:16:35.3151814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102329.xml (deflated 46%)
2022-12-01T11:16:35.3152592Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102336.xml (deflated 40%)
2022-12-01T11:16:35.3153348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102342.xml (deflated 41%)
2022-12-01T11:16:35.3154127Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102348.xml (deflated 40%)
2022-12-01T11:16:35.3154898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102354.xml (deflated 42%)
2022-12-01T11:16:35.3155790Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102400.xml (deflated 41%)
2022-12-01T11:16:35.3156574Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102407.xml (deflated 41%)
2022-12-01T11:16:35.3157360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102413.xml (deflated 41%)
2022-12-01T11:16:35.3158132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102419.xml (deflated 41%)
2022-12-01T11:16:35.3158905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102421.xml (deflated 42%)
2022-12-01T11:16:35.3159658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102423.xml (deflated 41%)
2022-12-01T11:16:35.3160540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102429.xml (deflated 42%)
2022-12-01T11:16:35.3161310Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102436.xml (deflated 42%)
2022-12-01T11:16:35.3162080Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102438.xml (deflated 43%)
2022-12-01T11:16:35.3162843Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102440.xml (deflated 42%)
2022-12-01T11:16:35.3163627Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102442.xml (deflated 42%)
2022-12-01T11:16:35.3164407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102444.xml (deflated 41%)
2022-12-01T11:16:35.3165199Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102450.xml (deflated 40%)
2022-12-01T11:16:35.3165958Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102457.xml (deflated 43%)
2022-12-01T11:16:35.3166795Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102459.xml (deflated 40%)
2022-12-01T11:16:35.3167629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102505.xml (deflated 41%)
2022-12-01T11:16:35.3168415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102511.xml (deflated 41%)
2022-12-01T11:16:35.3169189Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102517.xml (deflated 41%)
2022-12-01T11:16:35.3169950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102524.xml (deflated 42%)
2022-12-01T11:16:35.3170742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102530.xml (deflated 41%)
2022-12-01T11:16:35.3171517Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102536.xml (deflated 42%)
2022-12-01T11:16:35.3172291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102543.xml (deflated 41%)
2022-12-01T11:16:35.3173047Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102549.xml (deflated 40%)
2022-12-01T11:16:35.3173827Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102555.xml (deflated 41%)
2022-12-01T11:16:35.3174665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102601.xml (deflated 40%)
2022-12-01T11:16:35.3175465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102607.xml (deflated 41%)
2022-12-01T11:16:35.3176219Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102613.xml (deflated 41%)
2022-12-01T11:16:35.3176998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102620.xml (deflated 40%)
2022-12-01T11:16:35.3177779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102626.xml (deflated 40%)
2022-12-01T11:16:35.3178562Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102632.xml (deflated 41%)
2022-12-01T11:16:35.3179329Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102638.xml (deflated 41%)
2022-12-01T11:16:35.3180181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102645.xml (deflated 41%)
2022-12-01T11:16:35.3180965Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102651.xml (deflated 42%)
2022-12-01T11:16:35.3181737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102657.xml (deflated 42%)
2022-12-01T11:16:35.3182487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102704.xml (deflated 42%)
2022-12-01T11:16:35.3183265Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102710.xml (deflated 42%)
2022-12-01T11:16:35.3184041Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102716.xml (deflated 41%)
2022-12-01T11:16:35.3184819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102722.xml (deflated 41%)
2022-12-01T11:16:35.3185609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102728.xml (deflated 40%)
2022-12-01T11:16:35.3186361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102736.xml (deflated 41%)
2022-12-01T11:16:35.3187143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102745.xml (deflated 41%)
2022-12-01T11:16:35.3187915Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102751.xml (deflated 41%)
2022-12-01T11:16:35.3188683Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102759.xml (deflated 41%)
2022-12-01T11:16:35.3189844Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102805.xml (deflated 41%)
2022-12-01T11:16:35.3190648Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102811.xml (deflated 41%)
2022-12-01T11:16:35.3191428Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102817.xml (deflated 41%)
2022-12-01T11:16:35.3192200Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102827.xml (deflated 40%)
2022-12-01T11:16:35.3192954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102838.xml (deflated 41%)
2022-12-01T11:16:35.3193723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102848.xml (deflated 42%)
2022-12-01T11:16:35.3194584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102850.xml (deflated 42%)
2022-12-01T11:16:35.3195380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102852.xml (deflated 42%)
2022-12-01T11:16:35.3196143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102854.xml (deflated 41%)
2022-12-01T11:16:35.3196916Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102856.xml (deflated 42%)
2022-12-01T11:16:35.3197688Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102858.xml (deflated 42%)
2022-12-01T11:16:35.3198465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102900.xml (deflated 42%)
2022-12-01T11:16:35.3199222Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102902.xml (deflated 42%)
2022-12-01T11:16:35.3200090Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102904.xml (deflated 42%)
2022-12-01T11:16:35.3200861Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102906.xml (deflated 42%)
2022-12-01T11:16:35.3201641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102908.xml (deflated 43%)
2022-12-01T11:16:35.3202395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102910.xml (deflated 43%)
2022-12-01T11:16:35.3203172Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102913.xml (deflated 42%)
2022-12-01T11:16:35.3203954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102915.xml (deflated 42%)
2022-12-01T11:16:35.3204739Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102917.xml (deflated 42%)
2022-12-01T11:16:35.3205493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102919.xml (deflated 43%)
2022-12-01T11:16:35.3206266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102921.xml (deflated 43%)
2022-12-01T11:16:35.3207042Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102923.xml (deflated 42%)
2022-12-01T11:16:35.3207811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102925.xml (deflated 43%)
2022-12-01T11:16:35.3208569Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102927.xml (deflated 42%)
2022-12-01T11:16:35.3209354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102929.xml (deflated 42%)
2022-12-01T11:16:35.3210122Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102931.xml (deflated 42%)
2022-12-01T11:16:35.3211075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102933.xml (deflated 42%)
2022-12-01T11:16:35.3211828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102936.xml (deflated 43%)
2022-12-01T11:16:35.3212603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102937.xml (deflated 41%)
2022-12-01T11:16:35.3213370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102946.xml (deflated 42%)
2022-12-01T11:16:35.3214201Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102952.xml (deflated 42%)
2022-12-01T11:16:35.3214967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102955.xml (deflated 40%)
2022-12-01T11:16:35.3215737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103002.xml (deflated 41%)
2022-12-01T11:16:35.3216509Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103010.xml (deflated 40%)
2022-12-01T11:16:35.3217275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103018.xml (deflated 42%)
2022-12-01T11:16:35.3218020Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103024.xml (deflated 42%)
2022-12-01T11:16:35.3218872Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103030.xml (deflated 42%)
2022-12-01T11:16:35.3219642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103036.xml (deflated 40%)
2022-12-01T11:16:35.3220406Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103044.xml (deflated 42%)
2022-12-01T11:16:35.3221153Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103046.xml (deflated 42%)
2022-12-01T11:16:35.3221930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103052.xml (deflated 40%)
2022-12-01T11:16:35.3222697Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103058.xml (deflated 41%)
2022-12-01T11:16:35.3223475Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103104.xml (deflated 42%)
2022-12-01T11:16:35.3224231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103106.xml (deflated 41%)
2022-12-01T11:16:35.3225002Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103109.xml (deflated 41%)
2022-12-01T11:16:35.3225770Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103111.xml (deflated 41%)
2022-12-01T11:16:35.3226543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103113.xml (deflated 41%)
2022-12-01T11:16:35.3227312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103115.xml (deflated 41%)
2022-12-01T11:16:35.3228071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103117.xml (deflated 41%)
2022-12-01T11:16:35.3228854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103119.xml (deflated 42%)
2022-12-01T11:16:35.3230040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103121.xml (deflated 41%)
2022-12-01T11:16:35.3230883Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103127.xml (deflated 42%)
2022-12-01T11:16:35.3231647Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103131.xml (deflated 40%)
2022-12-01T11:16:35.3232401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103137.xml (deflated 42%)
2022-12-01T11:16:35.3233253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103144.xml (deflated 40%)
2022-12-01T11:16:35.3234055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103151.xml (deflated 42%)
2022-12-01T11:16:35.3234826Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103155.xml (deflated 42%)
2022-12-01T11:16:35.3235573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103159.xml (deflated 42%)
2022-12-01T11:16:35.3236349Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103203.xml (deflated 40%)
2022-12-01T11:16:35.3237114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103212.xml (deflated 40%)
2022-12-01T11:16:35.3237883Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103220.xml (deflated 42%)
2022-12-01T11:16:35.3238750Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103224.xml (deflated 42%)
2022-12-01T11:16:35.3239535Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103228.xml (deflated 40%)
2022-12-01T11:16:35.3240303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103236.xml (deflated 40%)
2022-12-01T11:16:35.3241073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103244.xml (deflated 40%)
2022-12-01T11:16:35.3241829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103252.xml (deflated 40%)
2022-12-01T11:16:35.3242602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103301.xml (deflated 42%)
2022-12-01T11:16:35.3243385Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103304.xml (deflated 41%)
2022-12-01T11:16:35.3244162Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103311.xml (deflated 42%)
2022-12-01T11:16:35.3244913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103315.xml (deflated 41%)
2022-12-01T11:16:35.3245690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103323.xml (deflated 42%)
2022-12-01T11:16:35.3246459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103327.xml (deflated 42%)
2022-12-01T11:16:35.3247226Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103331.xml (deflated 40%)
2022-12-01T11:16:35.3247984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103340.xml (deflated 40%)
2022-12-01T11:16:35.3248768Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103349.xml (deflated 42%)
2022-12-01T11:16:35.3249536Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103353.xml (deflated 41%)
2022-12-01T11:16:35.3250305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103401.xml (deflated 41%)
2022-12-01T11:16:35.3251054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103410.xml (deflated 41%)
2022-12-01T11:16:35.3251833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103418.xml (deflated 41%)
2022-12-01T11:16:35.3252661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103427.xml (deflated 41%)
2022-12-01T11:16:35.3253452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103446.xml (deflated 40%)
2022-12-01T11:16:35.3254203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103455.xml (deflated 40%)
2022-12-01T11:16:35.3254974Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103504.xml (deflated 40%)
2022-12-01T11:16:35.3255751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103513.xml (deflated 40%)
2022-12-01T11:16:35.3256519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103521.xml (deflated 40%)
2022-12-01T11:16:35.3257271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103530.xml (deflated 41%)
2022-12-01T11:16:35.3258123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103539.xml (deflated 41%)
2022-12-01T11:16:35.3258899Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103548.xml (deflated 41%)
2022-12-01T11:16:35.3259669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103556.xml (deflated 42%)
2022-12-01T11:16:35.3260435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103600.xml (deflated 41%)
2022-12-01T11:16:35.3261199Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103609.xml (deflated 40%)
2022-12-01T11:16:35.3261973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103617.xml (deflated 40%)
2022-12-01T11:16:35.3262751Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103623.xml (deflated 40%)
2022-12-01T11:16:35.3263529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103632.xml (deflated 40%)
2022-12-01T11:16:35.3264280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103640.xml (deflated 40%)
2022-12-01T11:16:35.3265053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103657.xml (deflated 41%)
2022-12-01T11:16:35.3265822Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103706.xml (deflated 41%)
2022-12-01T11:16:35.3266599Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103715.xml (deflated 42%)
2022-12-01T11:16:35.3267403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103723.xml (deflated 40%)
2022-12-01T11:16:35.3268191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103732.xml (deflated 42%)
2022-12-01T11:16:35.3268960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103736.xml (deflated 43%)
2022-12-01T11:16:35.3270072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103739.xml (deflated 42%)
2022-12-01T11:16:35.3270833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103748.xml (deflated 40%)
2022-12-01T11:16:35.3271601Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103756.xml (deflated 42%)
2022-12-01T11:16:35.3272452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103800.xml (deflated 41%)
2022-12-01T11:16:35.3273253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103809.xml (deflated 42%)
2022-12-01T11:16:35.3274007Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103812.xml (deflated 41%)
2022-12-01T11:16:35.3274780Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103822.xml (deflated 41%)
2022-12-01T11:16:35.3275553Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103830.xml (deflated 41%)
2022-12-01T11:16:35.3276323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103838.xml (deflated 42%)
2022-12-01T11:16:35.3277181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103842.xml (deflated 42%)
2022-12-01T11:16:35.3277964Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103846.xml (deflated 42%)
2022-12-01T11:16:35.3278736Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103850.xml (deflated 41%)
2022-12-01T11:16:35.3279507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103913.xml (deflated 40%)
2022-12-01T11:16:35.3280263Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103919.xml (deflated 41%)
2022-12-01T11:16:35.3281033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103925.xml (deflated 42%)
2022-12-01T11:16:35.3281808Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103929.xml (deflated 42%)
2022-12-01T11:16:35.3282588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103933.xml (deflated 40%)
2022-12-01T11:16:35.3283338Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103939.xml (deflated 41%)
2022-12-01T11:16:35.3284107Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103945.xml (deflated 40%)
2022-12-01T11:16:35.3284876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103952.xml (deflated 42%)
2022-12-01T11:16:35.3285644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103954.xml (deflated 40%)
2022-12-01T11:16:35.3286398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104000.xml (deflated 42%)
2022-12-01T11:16:35.3287177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104007.xml (deflated 41%)
2022-12-01T11:16:35.3287943Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104013.xml (deflated 42%)
2022-12-01T11:16:35.3288708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104017.xml (deflated 41%)
2022-12-01T11:16:35.3289460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104023.xml (deflated 40%)
2022-12-01T11:16:35.3290234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104029.xml (deflated 41%)
2022-12-01T11:16:35.3291054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104036.xml (deflated 40%)
2022-12-01T11:16:35.3291851Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104042.xml (deflated 40%)
2022-12-01T11:16:35.3292602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104048.xml (deflated 40%)
2022-12-01T11:16:35.3293367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104057.xml (deflated 41%)
2022-12-01T11:16:35.3294138Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104103.xml (deflated 41%)
2022-12-01T11:16:35.3294904Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104109.xml (deflated 41%)
2022-12-01T11:16:35.3295657Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104116.xml (deflated 41%)
2022-12-01T11:16:35.3296507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104122.xml (deflated 41%)
2022-12-01T11:16:35.3297273Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104147.xml (deflated 41%)
2022-12-01T11:16:35.3298052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104212.xml (deflated 42%)
2022-12-01T11:16:35.3298824Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104217.xml (deflated 40%)
2022-12-01T11:16:35.3299576Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104225.xml (deflated 40%)
2022-12-01T11:16:35.3300351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104232.xml (deflated 40%)
2022-12-01T11:16:35.3301132Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104238.xml (deflated 41%)
2022-12-01T11:16:35.3301910Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104244.xml (deflated 42%)
2022-12-01T11:16:35.3302667Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104246.xml (deflated 42%)
2022-12-01T11:16:35.3303437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104248.xml (deflated 42%)
2022-12-01T11:16:35.3304208Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104250.xml (deflated 42%)
2022-12-01T11:16:35.3304975Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104252.xml (deflated 42%)
2022-12-01T11:16:35.3305737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104254.xml (deflated 42%)
2022-12-01T11:16:35.3306508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104256.xml (deflated 42%)
2022-12-01T11:16:35.3307271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104258.xml (deflated 42%)
2022-12-01T11:16:35.3308043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104300.xml (deflated 40%)
2022-12-01T11:16:35.3308791Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104306.xml (deflated 40%)
2022-12-01T11:16:35.3310194Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104312.xml (deflated 41%)
2022-12-01T11:16:35.3311053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104314.xml (deflated 42%)
2022-12-01T11:16:35.3311845Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104316.xml (deflated 42%)
2022-12-01T11:16:35.3312595Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104320.xml (deflated 40%)
2022-12-01T11:16:35.3313371Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104328.xml (deflated 41%)
2022-12-01T11:16:35.3314142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104337.xml (deflated 40%)
2022-12-01T11:16:35.3314913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104345.xml (deflated 42%)
2022-12-01T11:16:35.3315672Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104349.xml (deflated 41%)
2022-12-01T11:16:35.3316534Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104353.xml (deflated 40%)
2022-12-01T11:16:35.3317301Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104359.xml (deflated 40%)
2022-12-01T11:16:35.3318066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104405.xml (deflated 42%)
2022-12-01T11:16:35.3318820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104408.xml (deflated 40%)
2022-12-01T11:16:35.3319585Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104414.xml (deflated 40%)
2022-12-01T11:16:35.3320356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104421.xml (deflated 41%)
2022-12-01T11:16:35.3321135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104427.xml (deflated 40%)
2022-12-01T11:16:35.3321889Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104433.xml (deflated 42%)
2022-12-01T11:16:35.3322658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104439.xml (deflated 42%)
2022-12-01T11:16:35.3323431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104445.xml (deflated 42%)
2022-12-01T11:16:35.3324196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104451.xml (deflated 42%)
2022-12-01T11:16:35.3324947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104458.xml (deflated 41%)
2022-12-01T11:16:35.3325829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104504.xml (deflated 41%)
2022-12-01T11:16:35.3326583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104511.xml (deflated 42%)
2022-12-01T11:16:35.3327354Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104513.xml (deflated 41%)
2022-12-01T11:16:35.3328119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104518.xml (deflated 43%)
2022-12-01T11:16:35.3328885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104520.xml (deflated 43%)
2022-12-01T11:16:35.3329628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104522.xml (deflated 40%)
2022-12-01T11:16:35.3330464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104528.xml (deflated 42%)
2022-12-01T11:16:35.3331250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104530.xml (deflated 42%)
2022-12-01T11:16:35.3332013Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104532.xml (deflated 40%)
2022-12-01T11:16:35.3332782Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104538.xml (deflated 41%)
2022-12-01T11:16:35.3333540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104543.xml (deflated 40%)
2022-12-01T11:16:35.3334308Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104549.xml (deflated 41%)
2022-12-01T11:16:35.3335143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104555.xml (deflated 42%)
2022-12-01T11:16:35.3335915Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104557.xml (deflated 41%)
2022-12-01T11:16:35.3336671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104559.xml (deflated 40%)
2022-12-01T11:16:35.3337438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104605.xml (deflated 42%)
2022-12-01T11:16:35.3338205Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104610.xml (deflated 41%)
2022-12-01T11:16:35.3338972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104616.xml (deflated 40%)
2022-12-01T11:16:35.3339728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104622.xml (deflated 41%)
2022-12-01T11:16:35.3340508Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104628.xml (deflated 40%)
2022-12-01T11:16:35.3341275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104634.xml (deflated 40%)
2022-12-01T11:16:35.3342044Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104640.xml (deflated 41%)
2022-12-01T11:16:35.3342794Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104645.xml (deflated 41%)
2022-12-01T11:16:35.3343568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104647.xml (deflated 41%)
2022-12-01T11:16:35.3344336Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104649.xml (deflated 41%)
2022-12-01T11:16:35.3345111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104651.xml (deflated 41%)
2022-12-01T11:16:35.3345859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104657.xml (deflated 41%)
2022-12-01T11:16:35.3346631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104703.xml (deflated 40%)
2022-12-01T11:16:35.3347401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104709.xml (deflated 40%)
2022-12-01T11:16:35.3348174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104714.xml (deflated 41%)
2022-12-01T11:16:35.3348977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104720.xml (deflated 41%)
2022-12-01T11:16:35.3350163Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104728.xml (deflated 41%)
2022-12-01T11:16:35.3350936Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104735.xml (deflated 41%)
2022-12-01T11:16:35.3351702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104741.xml (deflated 40%)
2022-12-01T11:16:35.3352452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104748.xml (deflated 41%)
2022-12-01T11:16:35.3353223Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104756.xml (deflated 40%)
2022-12-01T11:16:35.3353999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104808.xml (deflated 40%)
2022-12-01T11:16:35.3354876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104823.xml (deflated 41%)
2022-12-01T11:16:35.3355628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104832.xml (deflated 42%)
2022-12-01T11:16:35.3356400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104834.xml (deflated 42%)
2022-12-01T11:16:35.3357168Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104841.xml (deflated 43%)
2022-12-01T11:16:35.3357936Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104845.xml (deflated 42%)
2022-12-01T11:16:35.3358700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104851.xml (deflated 41%)
2022-12-01T11:16:35.3359476Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104857.xml (deflated 40%)
2022-12-01T11:16:35.3360247Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104906.xml (deflated 40%)
2022-12-01T11:16:35.3361012Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104915.xml (deflated 40%)
2022-12-01T11:16:35.3361760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104923.xml (deflated 39%)
2022-12-01T11:16:35.3362526Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104931.xml (deflated 39%)
2022-12-01T11:16:35.3363301Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104940.xml (deflated 40%)
2022-12-01T11:16:35.3364070Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104948.xml (deflated 40%)
2022-12-01T11:16:35.3364821Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104956.xml (deflated 42%)
2022-12-01T11:16:35.3365585Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105000.xml (deflated 42%)
2022-12-01T11:16:35.3366351Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105007.xml (deflated 42%)
2022-12-01T11:16:35.3367121Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105010.xml (deflated 42%)
2022-12-01T11:16:35.3367921Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105018.xml (deflated 42%)
2022-12-01T11:16:35.3368773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105021.xml (deflated 46%)
2022-12-01T11:16:35.3369566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105027.xml (deflated 47%)
2022-12-01T11:16:35.3370328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105033.xml (deflated 48%)
2022-12-01T11:16:35.3371075Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105039.xml (deflated 46%)
2022-12-01T11:16:35.3371842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105046.xml (deflated 40%)
2022-12-01T11:16:35.3372606Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105052.xml (deflated 41%)
2022-12-01T11:16:35.3373380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105058.xml (deflated 41%)
2022-12-01T11:16:35.3374233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105104.xml (deflated 42%)
2022-12-01T11:16:35.3374979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105110.xml (deflated 41%)
2022-12-01T11:16:35.3375748Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105117.xml (deflated 41%)
2022-12-01T11:16:35.3376516Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105123.xml (deflated 40%)
2022-12-01T11:16:35.3377280Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105130.xml (deflated 41%)
2022-12-01T11:16:35.3378038Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105132.xml (deflated 41%)
2022-12-01T11:16:35.3378814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105134.xml (deflated 41%)
2022-12-01T11:16:35.3379577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105140.xml (deflated 42%)
2022-12-01T11:16:35.3380340Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105146.xml (deflated 42%)
2022-12-01T11:16:35.3381083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105148.xml (deflated 43%)
2022-12-01T11:16:35.3381842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105150.xml (deflated 42%)
2022-12-01T11:16:35.3382611Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105152.xml (deflated 42%)
2022-12-01T11:16:35.3383386Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105154.xml (deflated 41%)
2022-12-01T11:16:35.3384136Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105201.xml (deflated 40%)
2022-12-01T11:16:35.3385010Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105207.xml (deflated 43%)
2022-12-01T11:16:35.3385777Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105209.xml (deflated 40%)
2022-12-01T11:16:35.3386548Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105216.xml (deflated 40%)
2022-12-01T11:16:35.3387294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105222.xml (deflated 41%)
2022-12-01T11:16:35.3388123Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105228.xml (deflated 40%)
2022-12-01T11:16:35.3388906Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105235.xml (deflated 42%)
2022-12-01T11:16:35.3390048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105241.xml (deflated 41%)
2022-12-01T11:16:35.3390801Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105247.xml (deflated 42%)
2022-12-01T11:16:35.3391566Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105253.xml (deflated 42%)
2022-12-01T11:16:35.3392325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105259.xml (deflated 40%)
2022-12-01T11:16:35.3393206Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105305.xml (deflated 41%)
2022-12-01T11:16:35.3393953Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105311.xml (deflated 41%)
2022-12-01T11:16:35.3394727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105318.xml (deflated 41%)
2022-12-01T11:16:35.3395494Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105324.xml (deflated 41%)
2022-12-01T11:16:35.3396250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105330.xml (deflated 41%)
2022-12-01T11:16:35.3396983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105336.xml (deflated 40%)
2022-12-01T11:16:35.3397741Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105343.xml (deflated 40%)
2022-12-01T11:16:35.3398512Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105349.xml (deflated 41%)
2022-12-01T11:16:35.3399279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105356.xml (deflated 40%)
2022-12-01T11:16:35.3400029Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105402.xml (deflated 42%)
2022-12-01T11:16:35.3400805Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105408.xml (deflated 42%)
2022-12-01T11:16:35.3401569Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105415.xml (deflated 41%)
2022-12-01T11:16:35.3402342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105421.xml (deflated 42%)
2022-12-01T11:16:35.3403094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105427.xml (deflated 41%)
2022-12-01T11:16:35.3403854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105433.xml (deflated 40%)
2022-12-01T11:16:35.3404693Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105439.xml (deflated 40%)
2022-12-01T11:16:35.3405464Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105448.xml (deflated 40%)
2022-12-01T11:16:35.3406210Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105456.xml (deflated 40%)
2022-12-01T11:16:35.3407053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105502.xml (deflated 41%)
2022-12-01T11:16:35.3407844Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105510.xml (deflated 41%)
2022-12-01T11:16:35.3408603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105516.xml (deflated 40%)
2022-12-01T11:16:35.3409347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105522.xml (deflated 41%)
2022-12-01T11:16:35.3410112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105528.xml (deflated 40%)
2022-12-01T11:16:35.3410869Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105538.xml (deflated 40%)
2022-12-01T11:16:35.3411638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105548.xml (deflated 41%)
2022-12-01T11:16:35.3412459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105558.xml (deflated 42%)
2022-12-01T11:16:35.3413236Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105600.xml (deflated 42%)
2022-12-01T11:16:35.3414000Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105602.xml (deflated 42%)
2022-12-01T11:16:35.3414767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105605.xml (deflated 41%)
2022-12-01T11:16:35.3415532Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105607.xml (deflated 42%)
2022-12-01T11:16:35.3416285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105609.xml (deflated 42%)
2022-12-01T11:16:35.3417060Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105611.xml (deflated 43%)
2022-12-01T11:16:35.3417837Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105613.xml (deflated 42%)
2022-12-01T11:16:35.3418609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105615.xml (deflated 42%)
2022-12-01T11:16:35.3419361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105617.xml (deflated 42%)
2022-12-01T11:16:35.3420128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105619.xml (deflated 43%)
2022-12-01T11:16:35.3420898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105621.xml (deflated 43%)
2022-12-01T11:16:35.3421671Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105623.xml (deflated 42%)
2022-12-01T11:16:35.3422424Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105625.xml (deflated 42%)
2022-12-01T11:16:35.3423191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105627.xml (deflated 42%)
2022-12-01T11:16:35.3423954Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105629.xml (deflated 43%)
2022-12-01T11:16:35.3424719Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105631.xml (deflated 42%)
2022-12-01T11:16:35.3425473Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105633.xml (deflated 42%)
2022-12-01T11:16:35.3426293Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105635.xml (deflated 42%)
2022-12-01T11:16:35.3427079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105637.xml (deflated 42%)
2022-12-01T11:16:35.3427853Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105639.xml (deflated 42%)
2022-12-01T11:16:35.3428600Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105642.xml (deflated 42%)
2022-12-01T11:16:35.3429749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105644.xml (deflated 42%)
2022-12-01T11:16:35.3430542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105646.xml (deflated 42%)
2022-12-01T11:16:35.3431317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105648.xml (deflated 41%)
2022-12-01T11:16:35.3432168Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105657.xml (deflated 42%)
2022-12-01T11:16:35.3432944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105703.xml (deflated 42%)
2022-12-01T11:16:35.3433710Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105705.xml (deflated 41%)
2022-12-01T11:16:35.3434478Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105712.xml (deflated 41%)
2022-12-01T11:16:35.3435229Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105721.xml (deflated 40%)
2022-12-01T11:16:35.3436005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105728.xml (deflated 42%)
2022-12-01T11:16:35.3436779Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105734.xml (deflated 42%)
2022-12-01T11:16:35.3437544Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105740.xml (deflated 42%)
2022-12-01T11:16:35.3438283Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105747.xml (deflated 40%)
2022-12-01T11:16:35.3439040Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105754.xml (deflated 40%)
2022-12-01T11:16:35.3439801Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105801.xml (deflated 42%)
2022-12-01T11:16:35.3440568Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105808.xml (deflated 40%)
2022-12-01T11:16:35.3441311Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105814.xml (deflated 41%)
2022-12-01T11:16:35.3442076Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105820.xml (deflated 42%)
2022-12-01T11:16:35.3442848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105822.xml (deflated 41%)
2022-12-01T11:16:35.3443608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105824.xml (deflated 40%)
2022-12-01T11:16:35.3444355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105826.xml (deflated 41%)
2022-12-01T11:16:35.3445130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105829.xml (deflated 41%)
2022-12-01T11:16:35.3445995Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105831.xml (deflated 41%)
2022-12-01T11:16:35.3446786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105833.xml (deflated 41%)
2022-12-01T11:16:35.3447543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105835.xml (deflated 41%)
2022-12-01T11:16:35.3448316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105837.xml (deflated 41%)
2022-12-01T11:16:35.3449079Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105843.xml (deflated 42%)
2022-12-01T11:16:35.3449852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105847.xml (deflated 40%)
2022-12-01T11:16:35.3450703Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105853.xml (deflated 42%)
2022-12-01T11:16:35.3451459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105859.xml (deflated 40%)
2022-12-01T11:16:35.3452231Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105907.xml (deflated 42%)
2022-12-01T11:16:35.3452992Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105911.xml (deflated 42%)
2022-12-01T11:16:35.3453759Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105915.xml (deflated 42%)
2022-12-01T11:16:35.3454517Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105919.xml (deflated 40%)
2022-12-01T11:16:35.3455288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105928.xml (deflated 40%)
2022-12-01T11:16:35.3456060Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105936.xml (deflated 42%)
2022-12-01T11:16:35.3456839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105940.xml (deflated 42%)
2022-12-01T11:16:35.3457588Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105944.xml (deflated 40%)
2022-12-01T11:16:35.3458362Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105952.xml (deflated 40%)
2022-12-01T11:16:35.3459130Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110000.xml (deflated 41%)
2022-12-01T11:16:35.3459900Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110008.xml (deflated 41%)
2022-12-01T11:16:35.3460655Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110017.xml (deflated 42%)
2022-12-01T11:16:35.3461417Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110020.xml (deflated 41%)
2022-12-01T11:16:35.3462180Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110027.xml (deflated 42%)
2022-12-01T11:16:35.3462949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110030.xml (deflated 41%)
2022-12-01T11:16:35.3463698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110039.xml (deflated 42%)
2022-12-01T11:16:35.3464501Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110043.xml (deflated 42%)
2022-12-01T11:16:35.3465266Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110047.xml (deflated 40%)
2022-12-01T11:16:35.3466010Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110056.xml (deflated 40%)
2022-12-01T11:16:35.3466747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110105.xml (deflated 42%)
2022-12-01T11:16:35.3467559Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110109.xml (deflated 41%)
2022-12-01T11:16:35.3468326Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110117.xml (deflated 41%)
2022-12-01T11:16:35.3469093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110125.xml (deflated 41%)
2022-12-01T11:16:35.3470440Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110134.xml (deflated 41%)
2022-12-01T11:16:35.3471221Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110153.xml (deflated 40%)
2022-12-01T11:16:35.3471990Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110202.xml (deflated 41%)
2022-12-01T11:16:35.3472757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110211.xml (deflated 41%)
2022-12-01T11:16:35.3473510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110219.xml (deflated 41%)
2022-12-01T11:16:35.3474283Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110228.xml (deflated 41%)
2022-12-01T11:16:35.3475053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110237.xml (deflated 41%)
2022-12-01T11:16:35.3475825Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110245.xml (deflated 41%)
2022-12-01T11:16:35.3476573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110254.xml (deflated 41%)
2022-12-01T11:16:35.3477347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110302.xml (deflated 41%)
2022-12-01T11:16:35.3478114Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110311.xml (deflated 42%)
2022-12-01T11:16:35.3478879Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110315.xml (deflated 42%)
2022-12-01T11:16:35.3479639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110324.xml (deflated 41%)
2022-12-01T11:16:35.3480410Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110332.xml (deflated 40%)
2022-12-01T11:16:35.3481174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110339.xml (deflated 40%)
2022-12-01T11:16:35.3481944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110347.xml (deflated 40%)
2022-12-01T11:16:35.3482714Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110355.xml (deflated 40%)
2022-12-01T11:16:35.3483469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110413.xml (deflated 41%)
2022-12-01T11:16:35.3484313Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110422.xml (deflated 41%)
2022-12-01T11:16:35.3485111Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110430.xml (deflated 41%)
2022-12-01T11:16:35.3485876Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110439.xml (deflated 40%)
2022-12-01T11:16:35.3486628Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110447.xml (deflated 42%)
2022-12-01T11:16:35.3487392Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110451.xml (deflated 43%)
2022-12-01T11:16:35.3488154Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110455.xml (deflated 42%)
2022-12-01T11:16:35.3488923Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110504.xml (deflated 41%)
2022-12-01T11:16:35.3489773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110512.xml (deflated 42%)
2022-12-01T11:16:35.3490539Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110516.xml (deflated 41%)
2022-12-01T11:16:35.3491296Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110524.xml (deflated 42%)
2022-12-01T11:16:35.3492067Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110528.xml (deflated 41%)
2022-12-01T11:16:35.3492819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110537.xml (deflated 41%)
2022-12-01T11:16:35.3493592Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110545.xml (deflated 41%)
2022-12-01T11:16:35.3494367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110553.xml (deflated 42%)
2022-12-01T11:16:35.3495135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110557.xml (deflated 42%)
2022-12-01T11:16:35.3495887Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110601.xml (deflated 42%)
2022-12-01T11:16:35.3496647Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110605.xml (deflated 41%)
2022-12-01T11:16:35.3497421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110619.xml (deflated 40%)
2022-12-01T11:16:35.3498192Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110625.xml (deflated 41%)
2022-12-01T11:16:35.3498950Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110632.xml (deflated 42%)
2022-12-01T11:16:35.3499726Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110635.xml (deflated 42%)
2022-12-01T11:16:35.3500492Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110639.xml (deflated 40%)
2022-12-01T11:16:35.3501259Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110646.xml (deflated 41%)
2022-12-01T11:16:35.3502004Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110652.xml (deflated 41%)
2022-12-01T11:16:35.3502773Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110658.xml (deflated 41%)
2022-12-01T11:16:35.3503593Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110700.xml (deflated 41%)
2022-12-01T11:16:35.3504379Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110706.xml (deflated 42%)
2022-12-01T11:16:35.3505127Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110712.xml (deflated 40%)
2022-12-01T11:16:35.3505902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110718.xml (deflated 42%)
2022-12-01T11:16:35.3506665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110722.xml (deflated 41%)
2022-12-01T11:16:35.3507437Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110729.xml (deflated 41%)
2022-12-01T11:16:35.3508270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110735.xml (deflated 40%)
2022-12-01T11:16:35.3509045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110741.xml (deflated 40%)
2022-12-01T11:16:35.3510212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110748.xml (deflated 40%)
2022-12-01T11:16:35.3510984Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110754.xml (deflated 40%)
2022-12-01T11:16:35.3511733Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110802.xml (deflated 41%)
2022-12-01T11:16:35.3512498Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110809.xml (deflated 41%)
2022-12-01T11:16:35.3513267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110815.xml (deflated 42%)
2022-12-01T11:16:35.3514037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110822.xml (deflated 41%)
2022-12-01T11:16:35.3514788Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110828.xml (deflated 41%)
2022-12-01T11:16:35.3515564Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110853.xml (deflated 41%)
2022-12-01T11:16:35.3516328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110918.xml (deflated 43%)
2022-12-01T11:16:35.3517099Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110924.xml (deflated 40%)
2022-12-01T11:16:35.3517852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110931.xml (deflated 40%)
2022-12-01T11:16:35.3518626Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110938.xml (deflated 40%)
2022-12-01T11:16:35.3519398Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110944.xml (deflated 41%)
2022-12-01T11:16:35.3520167Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110950.xml (deflated 42%)
2022-12-01T11:16:35.3520917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110952.xml (deflated 42%)
2022-12-01T11:16:35.3521690Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110954.xml (deflated 42%)
2022-12-01T11:16:35.3522540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110956.xml (deflated 42%)
2022-12-01T11:16:35.3523332Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110958.xml (deflated 42%)
2022-12-01T11:16:35.3524106Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111000.xml (deflated 42%)
2022-12-01T11:16:35.3524857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111002.xml (deflated 42%)
2022-12-01T11:16:35.3525629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111004.xml (deflated 42%)
2022-12-01T11:16:35.3526400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111006.xml (deflated 40%)
2022-12-01T11:16:35.3527171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111012.xml (deflated 41%)
2022-12-01T11:16:35.3528015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111018.xml (deflated 42%)
2022-12-01T11:16:35.3528784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111020.xml (deflated 43%)
2022-12-01T11:16:35.3529557Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111022.xml (deflated 42%)
2022-12-01T11:16:35.3530323Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111026.xml (deflated 41%)
2022-12-01T11:16:35.3531074Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111034.xml (deflated 41%)
2022-12-01T11:16:35.3531842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111042.xml (deflated 41%)
2022-12-01T11:16:35.3532618Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111051.xml (deflated 42%)
2022-12-01T11:16:35.3533388Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111055.xml (deflated 41%)
2022-12-01T11:16:35.3534142Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111058.xml (deflated 41%)
2022-12-01T11:16:35.3534918Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111104.xml (deflated 40%)
2022-12-01T11:16:35.3535684Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111110.xml (deflated 42%)
2022-12-01T11:16:35.3536455Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111114.xml (deflated 41%)
2022-12-01T11:16:35.3537214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111120.xml (deflated 41%)
2022-12-01T11:16:35.3537989Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111126.xml (deflated 41%)
2022-12-01T11:16:35.3538760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111132.xml (deflated 41%)
2022-12-01T11:16:35.3539530Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111138.xml (deflated 43%)
2022-12-01T11:16:35.3540279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111144.xml (deflated 43%)
2022-12-01T11:16:35.3541053Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111151.xml (deflated 43%)
2022-12-01T11:16:35.3541874Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111157.xml (deflated 43%)
2022-12-01T11:16:35.3542656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111204.xml (deflated 41%)
2022-12-01T11:16:35.3543407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111210.xml (deflated 41%)
2022-12-01T11:16:35.3544174Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111217.xml (deflated 43%)
2022-12-01T11:16:35.3544947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111219.xml (deflated 41%)
2022-12-01T11:16:35.3545711Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111225.xml (deflated 43%)
2022-12-01T11:16:35.3546463Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111227.xml (deflated 43%)
2022-12-01T11:16:35.3547303Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111229.xml (deflated 41%)
2022-12-01T11:16:35.3548071Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111234.xml (deflated 42%)
2022-12-01T11:16:35.3548842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111236.xml (deflated 42%)
2022-12-01T11:16:35.3549973Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111238.xml (deflated 41%)
2022-12-01T11:16:35.3550756Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111244.xml (deflated 41%)
2022-12-01T11:16:35.3551529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111250.xml (deflated 41%)
2022-12-01T11:16:35.3552305Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111256.xml (deflated 41%)
2022-12-01T11:16:35.3553055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111302.xml (deflated 42%)
2022-12-01T11:16:35.3553828Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111303.xml (deflated 41%)
2022-12-01T11:16:35.3554598Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111305.xml (deflated 41%)
2022-12-01T11:16:35.3555370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111311.xml (deflated 43%)
2022-12-01T11:16:35.3556112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111317.xml (deflated 41%)
2022-12-01T11:16:35.3556893Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111323.xml (deflated 41%)
2022-12-01T11:16:35.3557663Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111328.xml (deflated 41%)
2022-12-01T11:16:35.3558433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111334.xml (deflated 40%)
2022-12-01T11:16:35.3559183Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111340.xml (deflated 40%)
2022-12-01T11:16:35.3559961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111346.xml (deflated 41%)
2022-12-01T11:16:35.3560728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111351.xml (deflated 42%)
2022-12-01T11:16:35.3561578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111353.xml (deflated 41%)
2022-12-01T11:16:35.3562352Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111355.xml (deflated 41%)
2022-12-01T11:16:35.3563125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111357.xml (deflated 41%)
2022-12-01T11:16:35.3563894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111403.xml (deflated 41%)
2022-12-01T11:16:35.3564667Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111409.xml (deflated 40%)
2022-12-01T11:16:35.3565431Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111415.xml (deflated 40%)
2022-12-01T11:16:35.3566271Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111420.xml (deflated 41%)
2022-12-01T11:16:35.3567045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111426.xml (deflated 41%)
2022-12-01T11:16:35.3567859Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111434.xml (deflated 41%)
2022-12-01T11:16:35.3568634Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111441.xml (deflated 41%)
2022-12-01T11:16:35.3569390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111447.xml (deflated 40%)
2022-12-01T11:16:35.3570157Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111454.xml (deflated 41%)
2022-12-01T11:16:35.3570928Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111502.xml (deflated 40%)
2022-12-01T11:16:35.3571702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111514.xml (deflated 40%)
2022-12-01T11:16:35.3572468Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102113.xml (deflated 41%)
2022-12-01T11:16:35.3573252Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102123.xml (deflated 42%)
2022-12-01T11:16:35.3574034Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102125.xml (deflated 42%)
2022-12-01T11:16:35.3574820Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml (deflated 43%)
2022-12-01T11:16:35.3575586Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml (deflated 41%)
2022-12-01T11:16:35.3576373Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102143.xml (deflated 41%)
2022-12-01T11:16:35.3577154Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102151.xml (deflated 41%)
2022-12-01T11:16:35.3577931Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102201.xml (deflated 41%)
2022-12-01T11:16:35.3578690Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102209.xml (deflated 41%)
2022-12-01T11:16:35.3579481Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102218.xml (deflated 40%)
2022-12-01T11:16:35.3580308Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102226.xml (deflated 40%)
2022-12-01T11:16:35.3581101Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102236.xml (deflated 40%)
2022-12-01T11:16:35.3581857Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102245.xml (deflated 40%)
2022-12-01T11:16:35.3582636Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102253.xml (deflated 42%)
2022-12-01T11:16:35.3583412Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102257.xml (deflated 42%)
2022-12-01T11:16:35.3584190Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102303.xml (deflated 42%)
2022-12-01T11:16:35.3584949Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102307.xml (deflated 42%)
2022-12-01T11:16:35.3585795Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102315.xml (deflated 42%)
2022-12-01T11:16:35.3586566Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102317.xml (deflated 45%)
2022-12-01T11:16:35.3587338Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102319.xml (deflated 47%)
2022-12-01T11:16:35.3588099Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102321.xml (deflated 48%)
2022-12-01T11:16:35.3588874Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102324.xml (deflated 45%)
2022-12-01T11:16:35.3590102Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102326.xml (deflated 40%)
2022-12-01T11:16:35.3590901Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102335.xml (deflated 44%)
2022-12-01T11:16:35.3591656Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102337.xml (deflated 44%)
2022-12-01T11:16:35.3592434Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102339.xml (deflated 43%)
2022-12-01T11:16:35.3593205Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102342.xml (deflated 44%)
2022-12-01T11:16:35.3593973Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102344.xml (deflated 43%)
2022-12-01T11:16:35.3594730Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102346.xml (deflated 40%)
2022-12-01T11:16:35.3595505Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102355.xml (deflated 41%)
2022-12-01T11:16:35.3596282Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102357.xml (deflated 41%)
2022-12-01T11:16:35.3597056Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102359.xml (deflated 41%)
2022-12-01T11:16:35.3597813Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102409.xml (deflated 42%)
2022-12-01T11:16:35.3598587Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102415.xml (deflated 43%)
2022-12-01T11:16:35.3599365Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102417.xml (deflated 43%)
2022-12-01T11:16:35.3600215Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102419.xml (deflated 42%)
2022-12-01T11:16:35.3600986Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102421.xml (deflated 43%)
2022-12-01T11:16:35.3601751Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102423.xml (deflated 41%)
2022-12-01T11:16:35.3602521Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102425.xml (deflated 41%)
2022-12-01T11:16:35.3603289Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102427.xml (deflated 43%)
2022-12-01T11:16:35.3604033Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102429.xml (deflated 41%)
2022-12-01T11:16:35.3604815Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102431.xml (deflated 41%)
2022-12-01T11:16:35.3605686Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102434.xml (deflated 41%)
2022-12-01T11:16:35.3606458Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102436.xml (deflated 41%)
2022-12-01T11:16:35.3607230Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102438.xml (deflated 41%)
2022-12-01T11:16:35.3607987Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102440.xml (deflated 41%)
2022-12-01T11:16:35.3608767Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102442.xml (deflated 42%)
2022-12-01T11:16:35.3609540Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102444.xml (deflated 41%)
2022-12-01T11:16:35.3610322Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102446.xml (deflated 41%)
2022-12-01T11:16:35.3611078Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102448.xml (deflated 41%)
2022-12-01T11:16:35.3611843Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102455.xml (deflated 41%)
2022-12-01T11:16:35.3612612Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102457.xml (deflated 41%)
2022-12-01T11:16:35.3613384Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102459.xml (deflated 41%)
2022-12-01T11:16:35.3614139Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102501.xml (deflated 41%)
2022-12-01T11:16:35.3614921Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102507.xml (deflated 40%)
2022-12-01T11:16:35.3615687Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102516.xml (deflated 41%)
2022-12-01T11:16:35.3616457Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102526.xml (deflated 41%)
2022-12-01T11:16:35.3617212Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102535.xml (deflated 40%)
2022-12-01T11:16:35.3617983Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102544.xml (deflated 42%)
2022-12-01T11:16:35.3618751Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102551.xml (deflated 42%)
2022-12-01T11:16:35.3619578Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102557.xml (deflated 42%)
2022-12-01T11:16:35.3620352Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102603.xml (deflated 42%)
2022-12-01T11:16:35.3621120Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102609.xml (deflated 41%)
2022-12-01T11:16:35.3621883Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102618.xml (deflated 41%)
2022-12-01T11:16:35.3622653Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102628.xml (deflated 43%)
2022-12-01T11:16:35.3623410Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102630.xml (deflated 44%)
2022-12-01T11:16:35.3624195Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102632.xml (deflated 40%)
2022-12-01T11:16:35.3625029Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102641.xml (deflated 41%)
2022-12-01T11:16:35.3625798Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102643.xml (deflated 41%)
2022-12-01T11:16:35.3626549Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102653.xml (deflated 40%)
2022-12-01T11:16:35.3627321Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102702.xml (deflated 40%)
2022-12-01T11:16:35.3628092Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102712.xml (deflated 41%)
2022-12-01T11:16:35.3628867Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102714.xml (deflated 41%)
2022-12-01T11:16:35.3630010Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102716.xml (deflated 41%)
2022-12-01T11:16:35.3630797Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102718.xml (deflated 42%)
2022-12-01T11:16:35.3631569Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102720.xml (deflated 42%)
2022-12-01T11:16:35.3632338Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102722.xml (deflated 42%)
2022-12-01T11:16:35.3633090Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102724.xml (deflated 41%)
2022-12-01T11:16:35.3633866Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102726.xml (deflated 43%)
2022-12-01T11:16:35.3634643Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102728.xml (deflated 42%)
2022-12-01T11:16:35.3635417Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102730.xml (deflated 42%)
2022-12-01T11:16:35.3636166Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102732.xml (deflated 42%)
2022-12-01T11:16:35.3636935Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102735.xml (deflated 42%)
2022-12-01T11:16:35.3637703Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102737.xml (deflated 42%)
2022-12-01T11:16:35.3638470Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102739.xml (deflated 43%)
2022-12-01T11:16:35.3639300Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102741.xml (deflated 43%)
2022-12-01T11:16:35.3640105Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102743.xml (deflated 42%)
2022-12-01T11:16:35.3640875Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102745.xml (deflated 42%)
2022-12-01T11:16:35.3641641Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102747.xml (deflated 42%)
2022-12-01T11:16:35.3642393Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102750.xml (deflated 43%)
2022-12-01T11:16:35.3643168Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102752.xml (deflated 43%)
2022-12-01T11:16:35.3643941Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102754.xml (deflated 42%)
2022-12-01T11:16:35.3644818Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102756.xml (deflated 43%)
2022-12-01T11:16:35.3645583Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102758.xml (deflated 42%)
2022-12-01T11:16:35.3646332Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102800.xml (deflated 43%)
2022-12-01T11:16:35.3647105Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102802.xml (deflated 42%)
2022-12-01T11:16:35.3647868Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102804.xml (deflated 42%)
2022-12-01T11:16:35.3648641Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102806.xml (deflated 43%)
2022-12-01T11:16:35.3649402Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102808.xml (deflated 41%)
2022-12-01T11:16:35.3650171Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102818.xml (deflated 42%)
2022-12-01T11:16:35.3650937Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102820.xml (deflated 42%)
2022-12-01T11:16:35.3651708Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102822.xml (deflated 42%)
2022-12-01T11:16:35.3652463Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102824.xml (deflated 41%)
2022-12-01T11:16:35.3653228Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102826.xml (deflated 42%)
2022-12-01T11:16:35.3654007Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102828.xml (deflated 42%)
2022-12-01T11:16:35.3654784Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102834.xml (deflated 42%)
2022-12-01T11:16:35.3655531Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102836.xml (deflated 42%)
2022-12-01T11:16:35.3656287Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102842.xml (deflated 43%)
2022-12-01T11:16:35.3657055Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102845.xml (deflated 43%)
2022-12-01T11:16:35.3658074Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102847.xml (deflated 43%)
2022-12-01T11:16:35.3658892Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102849.xml (deflated 40%)
2022-12-01T11:16:35.3659683Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102851.xml (deflated 41%)
2022-12-01T11:16:35.3660450Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102853.xml (deflated 41%)
2022-12-01T11:16:35.3661221Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102855.xml (deflated 41%)
2022-12-01T11:16:35.3661975Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102857.xml (deflated 40%)
2022-12-01T11:16:35.3662747Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102859.xml (deflated 41%)
2022-12-01T11:16:35.3663525Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102901.xml (deflated 41%)
2022-12-01T11:16:35.3664371Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102903.xml (deflated 41%)
2022-12-01T11:16:35.3665120Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102905.xml (deflated 41%)
2022-12-01T11:16:35.3665891Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102907.xml (deflated 41%)
2022-12-01T11:16:35.3666661Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102909.xml (deflated 41%)
2022-12-01T11:16:35.3667428Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102919.xml (deflated 42%)
2022-12-01T11:16:35.3668229Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102921.xml (deflated 40%)
2022-12-01T11:16:35.3669009Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102930.xml (deflated 42%)
2022-12-01T11:16:35.3670136Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102936.xml (deflated 40%)
2022-12-01T11:16:35.3670893Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102945.xml (deflated 41%)
2022-12-01T11:16:35.3671634Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102947.xml (deflated 41%)
2022-12-01T11:16:35.3672390Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102949.xml (deflated 41%)
2022-12-01T11:16:35.3673153Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102951.xml (deflated 40%)
2022-12-01T11:16:35.3673916Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102959.xml (deflated 40%)
2022-12-01T11:16:35.3674664Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103008.xml (deflated 42%)
2022-12-01T11:16:35.3675416Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103012.xml (deflated 42%)
2022-12-01T11:16:35.3676174Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103016.xml (deflated 41%)
2022-12-01T11:16:35.3676925Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103018.xml (deflated 40%)
2022-12-01T11:16:35.3677668Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103027.xml (deflated 40%)
2022-12-01T11:16:35.3678502Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103035.xml (deflated 41%)
2022-12-01T11:16:35.3679300Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103037.xml (deflated 41%)
2022-12-01T11:16:35.3680067Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103040.xml (deflated 41%)
2022-12-01T11:16:35.3680822Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103048.xml (deflated 41%)
2022-12-01T11:16:35.3681603Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103050.xml (deflated 41%)
2022-12-01T11:16:35.3682372Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103052.xml (deflated 42%)
2022-12-01T11:16:35.3683229Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103056.xml (deflated 42%)
2022-12-01T11:16:35.3684005Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103100.xml (deflated 40%)
2022-12-01T11:16:35.3684752Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103109.xml (deflated 40%)
2022-12-01T11:16:35.3685514Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103118.xml (deflated 42%)
2022-12-01T11:16:35.3686281Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103122.xml (deflated 40%)
2022-12-01T11:16:35.3687216Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103124.xml (deflated 42%)
2022-12-01T11:16:35.3687971Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103126.xml (deflated 42%)
2022-12-01T11:16:35.3688754Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103128.xml (deflated 42%)
2022-12-01T11:16:35.3689525Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103130.xml (deflated 41%)
2022-12-01T11:16:35.3690290Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103132.xml (deflated 41%)
2022-12-01T11:16:35.3691041Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103134.xml (deflated 41%)
2022-12-01T11:16:35.3691810Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103137.xml (deflated 41%)
2022-12-01T11:16:35.3692578Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103139.xml (deflated 41%)
2022-12-01T11:16:35.3693348Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103141.xml (deflated 41%)
2022-12-01T11:16:35.3694102Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103143.xml (deflated 42%)
2022-12-01T11:16:35.3694857Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103145.xml (deflated 42%)
2022-12-01T11:16:35.3695611Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103147.xml (deflated 42%)
2022-12-01T11:16:35.3696365Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103149.xml (deflated 41%)
2022-12-01T11:16:35.3697168Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103151.xml (deflated 41%)
2022-12-01T11:16:35.3697967Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103200.xml (deflated 41%)
2022-12-01T11:16:35.3698741Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103209.xml (deflated 40%)
2022-12-01T11:16:35.3699507Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103217.xml (deflated 40%)
2022-12-01T11:16:35.3700255Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103226.xml (deflated 40%)
2022-12-01T11:16:35.3701028Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103228.xml (deflated 41%)
2022-12-01T11:16:35.3701797Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103230.xml (deflated 41%)
2022-12-01T11:16:35.3702628Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103232.xml (deflated 41%)
2022-12-01T11:16:35.3703379Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103234.xml (deflated 41%)
2022-12-01T11:16:35.3704147Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103237.xml (deflated 40%)
2022-12-01T11:16:35.3704969Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103245.xml (deflated 42%)
2022-12-01T11:16:35.3705733Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103249.xml (deflated 42%)
2022-12-01T11:16:35.3706498Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103251.xml (deflated 42%)
2022-12-01T11:16:35.3707264Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103253.xml (deflated 40%)
2022-12-01T11:16:35.3708020Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103302.xml (deflated 42%)
2022-12-01T11:16:35.3708784Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103306.xml (deflated 41%)
2022-12-01T11:16:35.3709914Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103308.xml (deflated 42%)
2022-12-01T11:16:35.3710682Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103312.xml (deflated 41%)
2022-12-01T11:16:35.3711449Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103322.xml (deflated 41%)
2022-12-01T11:16:35.3712220Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103330.xml (deflated 41%)
2022-12-01T11:16:35.3712991Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103338.xml (deflated 42%)
2022-12-01T11:16:35.3713747Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103342.xml (deflated 42%)
2022-12-01T11:16:35.3714519Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103346.xml (deflated 42%)
2022-12-01T11:16:35.3715280Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103350.xml (deflated 41%)
2022-12-01T11:16:35.3716048Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103352.xml (deflated 41%)
2022-12-01T11:16:35.3716876Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103359.xml (deflated 40%)
2022-12-01T11:16:35.3717672Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103405.xml (deflated 42%)
2022-12-01T11:16:35.3718442Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103409.xml (deflated 41%)
2022-12-01T11:16:35.3719219Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103411.xml (deflated 40%)
2022-12-01T11:16:35.3719970Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103417.xml (deflated 41%)
2022-12-01T11:16:35.3720738Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103419.xml (deflated 41%)
2022-12-01T11:16:35.3721519Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103422.xml (deflated 42%)
2022-12-01T11:16:35.3722372Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103424.xml (deflated 41%)
2022-12-01T11:16:35.3723124Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103426.xml (deflated 41%)
2022-12-01T11:16:35.3723893Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103428.xml (deflated 41%)
2022-12-01T11:16:35.3724659Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103430.xml (deflated 40%)
2022-12-01T11:16:35.3725428Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103432.xml (deflated 41%)
2022-12-01T11:16:35.3726178Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103448.xml (deflated 42%)
2022-12-01T11:16:35.3726955Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103450.xml (deflated 41%)
2022-12-01T11:16:35.3727726Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103456.xml (deflated 40%)
2022-12-01T11:16:35.3728497Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103502.xml (deflated 40%)
2022-12-01T11:16:35.3729245Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103509.xml (deflated 41%)
2022-12-01T11:16:35.3730012Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103511.xml (deflated 41%)
2022-12-01T11:16:35.3730780Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103519.xml (deflated 41%)
2022-12-01T11:16:35.3731558Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103528.xml (deflated 41%)
2022-12-01T11:16:35.3732307Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103537.xml (deflated 40%)
2022-12-01T11:16:35.3733078Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103546.xml (deflated 40%)
2022-12-01T11:16:35.3733847Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103549.xml (deflated 40%)
2022-12-01T11:16:35.3734611Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103551.xml (deflated 42%)
2022-12-01T11:16:35.3735362Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103553.xml (deflated 41%)
2022-12-01T11:16:35.3736185Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103555.xml (deflated 41%)
2022-12-01T11:16:35.3736969Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103557.xml (deflated 41%)
2022-12-01T11:16:35.3737735Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103559.xml (deflated 41%)
2022-12-01T11:16:35.3738488Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103601.xml (deflated 42%)
2022-12-01T11:16:35.3739251Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103603.xml (deflated 42%)
2022-12-01T11:16:35.3740017Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103605.xml (deflated 42%)
2022-12-01T11:16:35.3740790Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103607.xml (deflated 42%)
2022-12-01T11:16:35.3741611Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103609.xml (deflated 42%)
2022-12-01T11:16:35.3742382Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103611.xml (deflated 42%)
2022-12-01T11:16:35.3743152Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103614.xml (deflated 42%)
2022-12-01T11:16:35.3743914Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103616.xml (deflated 41%)
2022-12-01T11:16:35.3744679Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103618.xml (deflated 40%)
2022-12-01T11:16:35.3745433Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103624.xml (deflated 40%)
2022-12-01T11:16:35.3746205Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103630.xml (deflated 41%)
2022-12-01T11:16:35.3746975Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103632.xml (deflated 42%)
2022-12-01T11:16:35.3747745Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103635.xml (deflated 42%)
2022-12-01T11:16:35.3748489Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103639.xml (deflated 40%)
2022-12-01T11:16:35.3749258Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103647.xml (deflated 40%)
2022-12-01T11:16:35.3750409Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103656.xml (deflated 40%)
2022-12-01T11:16:35.3751183Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103706.xml (deflated 42%)
2022-12-01T11:16:35.3751940Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103710.xml (deflated 41%)
2022-12-01T11:16:35.3752701Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103714.xml (deflated 40%)
2022-12-01T11:16:35.3753466Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103720.xml (deflated 40%)
2022-12-01T11:16:35.3754236Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103726.xml (deflated 42%)
2022-12-01T11:16:35.3754985Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103730.xml (deflated 41%)
2022-12-01T11:16:35.3755839Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103740.xml (deflated 41%)
2022-12-01T11:16:35.3756627Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103749.xml (deflated 40%)
2022-12-01T11:16:35.3757400Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103758.xml (deflated 40%)
2022-12-01T11:16:35.3758149Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103808.xml (deflated 42%)
2022-12-01T11:16:35.3758919Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103814.xml (deflated 42%)
2022-12-01T11:16:35.3759681Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103820.xml (deflated 42%)
2022-12-01T11:16:35.3760451Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103826.xml (deflated 42%)
2022-12-01T11:16:35.3761294Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103833.xml (deflated 41%)
2022-12-01T11:16:35.3762061Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103842.xml (deflated 41%)
2022-12-01T11:16:35.3762827Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103852.xml (deflated 42%)
2022-12-01T11:16:35.3763596Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103854.xml (deflated 41%)
2022-12-01T11:16:35.3764348Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103903.xml (deflated 43%)
2022-12-01T11:16:35.3765121Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103905.xml (deflated 43%)
2022-12-01T11:16:35.3765894Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103907.xml (deflated 41%)
2022-12-01T11:16:35.3766665Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103916.xml (deflated 42%)
2022-12-01T11:16:35.3767417Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103918.xml (deflated 42%)
2022-12-01T11:16:35.3768232Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103920.xml (deflated 41%)
2022-12-01T11:16:35.3768997Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103929.xml (deflated 41%)
2022-12-01T11:16:35.3769767Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103931.xml (deflated 41%)
2022-12-01T11:16:35.3770519Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103933.xml (deflated 41%)
2022-12-01T11:16:35.3771277Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103936.xml (deflated 42%)
2022-12-01T11:16:35.3772038Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103938.xml (deflated 41%)
2022-12-01T11:16:35.3772804Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103940.xml (deflated 41%)
2022-12-01T11:16:35.3773554Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103942.xml (deflated 41%)
2022-12-01T11:16:35.3774334Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103944.xml (deflated 41%)
2022-12-01T11:16:35.3775160Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103946.xml (deflated 41%)
2022-12-01T11:16:35.3775941Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103954.xml (deflated 42%)
2022-12-01T11:16:35.3776692Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103956.xml (deflated 42%)
2022-12-01T11:16:35.3777462Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103958.xml (deflated 42%)
2022-12-01T11:16:35.3778228Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104000.xml (deflated 41%)
2022-12-01T11:16:35.3778999Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104010.xml (deflated 41%)
2022-12-01T11:16:35.3779834Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104012.xml (deflated 41%)
2022-12-01T11:16:35.3780602Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104014.xml (deflated 41%)
2022-12-01T11:16:35.3781372Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104016.xml (deflated 41%)
2022-12-01T11:16:35.3782139Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104025.xml (deflated 41%)
2022-12-01T11:16:35.3782907Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104034.xml (deflated 41%)
2022-12-01T11:16:35.3783659Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104043.xml (deflated 40%)
2022-12-01T11:16:35.3784433Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104052.xml (deflated 42%)
2022-12-01T11:16:35.3785208Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104054.xml (deflated 42%)
2022-12-01T11:16:35.3785974Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104056.xml (deflated 41%)
2022-12-01T11:16:35.3786721Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104105.xml (deflated 41%)
2022-12-01T11:16:35.3787493Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104114.xml (deflated 42%)
2022-12-01T11:16:35.3788248Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104116.xml (deflated 41%)
2022-12-01T11:16:35.3789015Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104125.xml (deflated 40%)
2022-12-01T11:16:35.3790171Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104127.xml (deflated 41%)
2022-12-01T11:16:35.3790942Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104823.xml (deflated 41%)
2022-12-01T11:16:35.3791706Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104833.xml (deflated 42%)
2022-12-01T11:16:35.3792474Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104835.xml (deflated 42%)
2022-12-01T11:16:35.3793230Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104841.xml (deflated 42%)
2022-12-01T11:16:35.3794004Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104845.xml (deflated 41%)
2022-12-01T11:16:35.3794856Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104853.xml (deflated 41%)
2022-12-01T11:16:35.3795653Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104901.xml (deflated 40%)
2022-12-01T11:16:35.3796404Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104911.xml (deflated 40%)
2022-12-01T11:16:35.3797172Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104919.xml (deflated 40%)
2022-12-01T11:16:35.3797936Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104928.xml (deflated 39%)
2022-12-01T11:16:35.3798708Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104937.xml (deflated 39%)
2022-12-01T11:16:35.3799555Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104946.xml (deflated 40%)
2022-12-01T11:16:35.3800336Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104955.xml (deflated 40%)
2022-12-01T11:16:35.3801105Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105003.xml (deflated 42%)
2022-12-01T11:16:35.3801877Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105007.xml (deflated 42%)
2022-12-01T11:16:35.3802631Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105013.xml (deflated 42%)
2022-12-01T11:16:35.3803401Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105017.xml (deflated 42%)
2022-12-01T11:16:35.3804177Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105026.xml (deflated 42%)
2022-12-01T11:16:35.3804951Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105028.xml (deflated 45%)
2022-12-01T11:16:35.3805707Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105030.xml (deflated 47%)
2022-12-01T11:16:35.3806480Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105032.xml (deflated 48%)
2022-12-01T11:16:35.3807248Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105034.xml (deflated 45%)
2022-12-01T11:16:35.3808018Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105036.xml (deflated 40%)
2022-12-01T11:16:35.3808776Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105046.xml (deflated 43%)
2022-12-01T11:16:35.3809553Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105048.xml (deflated 44%)
2022-12-01T11:16:35.3810325Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105050.xml (deflated 44%)
2022-12-01T11:16:35.3811091Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105052.xml (deflated 44%)
2022-12-01T11:16:35.3811842Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105054.xml (deflated 43%)
2022-12-01T11:16:35.3812615Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105056.xml (deflated 40%)
2022-12-01T11:16:35.3813435Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105106.xml (deflated 41%)
2022-12-01T11:16:35.3814222Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105108.xml (deflated 41%)
2022-12-01T11:16:35.3814975Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105110.xml (deflated 41%)
2022-12-01T11:16:35.3815748Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105119.xml (deflated 42%)
2022-12-01T11:16:35.3816511Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105125.xml (deflated 43%)
2022-12-01T11:16:35.3817276Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105127.xml (deflated 43%)
2022-12-01T11:16:35.3818027Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105130.xml (deflated 42%)
2022-12-01T11:16:35.3818880Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105132.xml (deflated 42%)
2022-12-01T11:16:35.3819647Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105134.xml (deflated 41%)
2022-12-01T11:16:35.3820420Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105136.xml (deflated 40%)
2022-12-01T11:16:35.3821169Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105138.xml (deflated 43%)
2022-12-01T11:16:35.3821943Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105140.xml (deflated 41%)
2022-12-01T11:16:35.3822714Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105142.xml (deflated 41%)
2022-12-01T11:16:35.3823483Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105144.xml (deflated 41%)
2022-12-01T11:16:35.3824250Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105146.xml (deflated 41%)
2022-12-01T11:16:35.3825015Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105148.xml (deflated 41%)
2022-12-01T11:16:35.3825787Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105150.xml (deflated 41%)
2022-12-01T11:16:35.3826552Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105152.xml (deflated 42%)
2022-12-01T11:16:35.3827316Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105154.xml (deflated 41%)
2022-12-01T11:16:35.3828073Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105156.xml (deflated 41%)
2022-12-01T11:16:35.3828846Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105159.xml (deflated 41%)
2022-12-01T11:16:35.3829986Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105205.xml (deflated 41%)
2022-12-01T11:16:35.3830768Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105207.xml (deflated 42%)
2022-12-01T11:16:35.3831518Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105209.xml (deflated 41%)
2022-12-01T11:16:35.3832282Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105211.xml (deflated 41%)
2022-12-01T11:16:35.3833124Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105218.xml (deflated 40%)
2022-12-01T11:16:35.3833919Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105227.xml (deflated 40%)
2022-12-01T11:16:35.3834671Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105236.xml (deflated 41%)
2022-12-01T11:16:35.3835447Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105246.xml (deflated 40%)
2022-12-01T11:16:35.3836208Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105255.xml (deflated 42%)
2022-12-01T11:16:35.3836971Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105301.xml (deflated 42%)
2022-12-01T11:16:35.3837729Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105307.xml (deflated 42%)
2022-12-01T11:16:35.3838588Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105314.xml (deflated 42%)
2022-12-01T11:16:35.3839357Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105320.xml (deflated 40%)
2022-12-01T11:16:35.3840132Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105329.xml (deflated 40%)
2022-12-01T11:16:35.3840880Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105338.xml (deflated 43%)
2022-12-01T11:16:35.3841646Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105340.xml (deflated 43%)
2022-12-01T11:16:35.3842410Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105343.xml (deflated 40%)
2022-12-01T11:16:35.3843186Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105352.xml (deflated 41%)
2022-12-01T11:16:35.3843938Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105354.xml (deflated 41%)
2022-12-01T11:16:35.3844696Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105403.xml (deflated 40%)
2022-12-01T11:16:35.3845452Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105413.xml (deflated 40%)
2022-12-01T11:16:35.3846208Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105422.xml (deflated 41%)
2022-12-01T11:16:35.3846959Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105424.xml (deflated 41%)
2022-12-01T11:16:35.3847737Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105426.xml (deflated 40%)
2022-12-01T11:16:35.3848505Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105428.xml (deflated 42%)
2022-12-01T11:16:35.3849265Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105430.xml (deflated 42%)
2022-12-01T11:16:35.3850016Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105433.xml (deflated 42%)
2022-12-01T11:16:35.3850773Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105435.xml (deflated 41%)
2022-12-01T11:16:35.3851539Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105437.xml (deflated 42%)
2022-12-01T11:16:35.3852356Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105439.xml (deflated 42%)
2022-12-01T11:16:35.3853127Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105441.xml (deflated 42%)
2022-12-01T11:16:35.3853898Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105443.xml (deflated 42%)
2022-12-01T11:16:35.3854665Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105445.xml (deflated 42%)
2022-12-01T11:16:35.3855435Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105447.xml (deflated 42%)
2022-12-01T11:16:35.3856182Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105449.xml (deflated 42%)
2022-12-01T11:16:35.3856957Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105452.xml (deflated 43%)
2022-12-01T11:16:35.3857791Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105454.xml (deflated 42%)
2022-12-01T11:16:35.3858557Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105456.xml (deflated 42%)
2022-12-01T11:16:35.3859305Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105458.xml (deflated 42%)
2022-12-01T11:16:35.3860079Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105500.xml (deflated 43%)
2022-12-01T11:16:35.3860847Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105502.xml (deflated 43%)
2022-12-01T11:16:35.3861628Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105504.xml (deflated 42%)
2022-12-01T11:16:35.3862396Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105506.xml (deflated 42%)
2022-12-01T11:16:35.3863145Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105508.xml (deflated 42%)
2022-12-01T11:16:35.3863905Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105510.xml (deflated 42%)
2022-12-01T11:16:35.3864676Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105512.xml (deflated 42%)
2022-12-01T11:16:35.3865437Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105514.xml (deflated 42%)
2022-12-01T11:16:35.3866186Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105517.xml (deflated 42%)
2022-12-01T11:16:35.3866961Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105519.xml (deflated 41%)
2022-12-01T11:16:35.3867771Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105528.xml (deflated 42%)
2022-12-01T11:16:35.3868542Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105530.xml (deflated 41%)
2022-12-01T11:16:35.3869298Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105532.xml (deflated 42%)
2022-12-01T11:16:35.3870443Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105535.xml (deflated 41%)
2022-12-01T11:16:35.3871203Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105537.xml (deflated 42%)
2022-12-01T11:16:35.3872053Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105539.xml (deflated 42%)
2022-12-01T11:16:35.3872829Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105545.xml (deflated 42%)
2022-12-01T11:16:35.3873598Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105547.xml (deflated 42%)
2022-12-01T11:16:35.3874364Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105553.xml (deflated 42%)
2022-12-01T11:16:35.3875131Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105556.xml (deflated 43%)
2022-12-01T11:16:35.3875885Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105558.xml (deflated 43%)
2022-12-01T11:16:35.3876656Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105600.xml (deflated 41%)
2022-12-01T11:16:35.3877510Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105602.xml (deflated 41%)
2022-12-01T11:16:35.3878274Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105604.xml (deflated 41%)
2022-12-01T11:16:35.3879025Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105606.xml (deflated 41%)
2022-12-01T11:16:35.3879865Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105608.xml (deflated 40%)
2022-12-01T11:16:35.3880624Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105610.xml (deflated 41%)
2022-12-01T11:16:35.3881396Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105612.xml (deflated 41%)
2022-12-01T11:16:35.3882158Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105614.xml (deflated 40%)
2022-12-01T11:16:35.3882917Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105616.xml (deflated 41%)
2022-12-01T11:16:35.3883676Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105618.xml (deflated 41%)
2022-12-01T11:16:35.3884441Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105620.xml (deflated 41%)
2022-12-01T11:16:35.3885188Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105630.xml (deflated 41%)
2022-12-01T11:16:35.3885953Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105632.xml (deflated 40%)
2022-12-01T11:16:35.3886723Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105641.xml (deflated 42%)
2022-12-01T11:16:35.3887484Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105648.xml (deflated 40%)
2022-12-01T11:16:35.3888234Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105656.xml (deflated 41%)
2022-12-01T11:16:35.3888995Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105658.xml (deflated 40%)
2022-12-01T11:16:35.3889763Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105700.xml (deflated 41%)
2022-12-01T11:16:35.3890521Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105702.xml (deflated 40%)
2022-12-01T11:16:35.3891337Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105711.xml (deflated 40%)
2022-12-01T11:16:35.3892109Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105720.xml (deflated 42%)
2022-12-01T11:16:35.3892861Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105724.xml (deflated 42%)
2022-12-01T11:16:35.3893614Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105728.xml (deflated 41%)
2022-12-01T11:16:35.3894366Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105730.xml (deflated 40%)
2022-12-01T11:16:35.3895132Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105738.xml (deflated 41%)
2022-12-01T11:16:35.3895963Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105747.xml (deflated 41%)
2022-12-01T11:16:35.3896719Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105749.xml (deflated 41%)
2022-12-01T11:16:35.3897472Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105751.xml (deflated 40%)
2022-12-01T11:16:35.3898239Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105800.xml (deflated 41%)
2022-12-01T11:16:35.3899004Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105802.xml (deflated 41%)
2022-12-01T11:16:35.3899762Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105804.xml (deflated 42%)
2022-12-01T11:16:35.3900515Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105808.xml (deflated 42%)
2022-12-01T11:16:35.3901285Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105812.xml (deflated 40%)
2022-12-01T11:16:35.3902041Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105821.xml (deflated 40%)
2022-12-01T11:16:35.3902806Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105830.xml (deflated 42%)
2022-12-01T11:16:35.3903560Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105834.xml (deflated 40%)
2022-12-01T11:16:35.3904326Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105836.xml (deflated 42%)
2022-12-01T11:16:35.3905091Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105838.xml (deflated 42%)
2022-12-01T11:16:35.3905861Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105840.xml (deflated 42%)
2022-12-01T11:16:35.3906304Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105842.xml (deflated 41%)
2022-12-01T11:16:35.3906746Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105844.xml (deflated 41%)
2022-12-01T11:16:35.3907166Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105846.xml (deflated 41%)
2022-12-01T11:16:35.3907605Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105848.xml (deflated 41%)
2022-12-01T11:16:35.3908090Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105850.xml (deflated 41%)
2022-12-01T11:16:35.3908549Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105853.xml (deflated 41%)
2022-12-01T11:16:35.3908985Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105855.xml (deflated 41%)
2022-12-01T11:16:35.3909798Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105857.xml (deflated 41%)
2022-12-01T11:16:35.3910250Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105859.xml (deflated 42%)
2022-12-01T11:16:35.3910687Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105901.xml (deflated 41%)
2022-12-01T11:16:35.3911120Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105903.xml (deflated 40%)
2022-12-01T11:16:35.3911670Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105912.xml (deflated 40%)
2022-12-01T11:16:35.3912106Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105921.xml (deflated 40%)
2022-12-01T11:16:35.3912524Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105929.xml (deflated 41%)
2022-12-01T11:16:35.3912961Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105938.xml (deflated 40%)
2022-12-01T11:16:35.3913395Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105940.xml (deflated 40%)
2022-12-01T11:16:35.3913830Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105942.xml (deflated 41%)
2022-12-01T11:16:35.3914271Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105944.xml (deflated 41%)
2022-12-01T11:16:35.3914708Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105946.xml (deflated 41%)
2022-12-01T11:16:35.3915144Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105948.xml (deflated 40%)
2022-12-01T11:16:35.3915579Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105957.xml (deflated 42%)
2022-12-01T11:16:35.3916012Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110001.xml (deflated 42%)
2022-12-01T11:16:35.3916447Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110004.xml (deflated 42%)
2022-12-01T11:16:35.3916867Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110006.xml (deflated 41%)
2022-12-01T11:16:35.3917308Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110014.xml (deflated 42%)
2022-12-01T11:16:35.3917742Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110018.xml (deflated 41%)
2022-12-01T11:16:35.3918174Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110020.xml (deflated 42%)
2022-12-01T11:16:35.3918646Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110024.xml (deflated 41%)
2022-12-01T11:16:35.3919075Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110034.xml (deflated 40%)
2022-12-01T11:16:35.3919569Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110042.xml (deflated 41%)
2022-12-01T11:16:35.3920025Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110050.xml (deflated 42%)
2022-12-01T11:16:35.3920454Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110054.xml (deflated 42%)
2022-12-01T11:16:35.3920885Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110058.xml (deflated 42%)
2022-12-01T11:16:35.3921302Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110102.xml (deflated 41%)
2022-12-01T11:16:35.3921735Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110104.xml (deflated 41%)
2022-12-01T11:16:35.3922164Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110110.xml (deflated 41%)
2022-12-01T11:16:35.3922670Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110117.xml (deflated 42%)
2022-12-01T11:16:35.3923104Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110121.xml (deflated 41%)
2022-12-01T11:16:35.3923535Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110123.xml (deflated 40%)
2022-12-01T11:16:35.3923965Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110129.xml (deflated 41%)
2022-12-01T11:16:35.3924393Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110132.xml (deflated 41%)
2022-12-01T11:16:35.3924821Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110134.xml (deflated 42%)
2022-12-01T11:16:35.3925271Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110136.xml (deflated 41%)
2022-12-01T11:16:35.3925704Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110138.xml (deflated 42%)
2022-12-01T11:16:35.3926120Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110140.xml (deflated 41%)
2022-12-01T11:16:35.3926552Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110142.xml (deflated 41%)
2022-12-01T11:16:35.3926984Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110144.xml (deflated 41%)
2022-12-01T11:16:35.3927418Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110156.xml (deflated 42%)
2022-12-01T11:16:35.3927857Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110158.xml (deflated 41%)
2022-12-01T11:16:35.3928289Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110204.xml (deflated 41%)
2022-12-01T11:16:35.3928721Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110211.xml (deflated 41%)
2022-12-01T11:16:35.3929157Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110217.xml (deflated 41%)
2022-12-01T11:16:35.3929591Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110219.xml (deflated 41%)
2022-12-01T11:16:35.3930035Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110228.xml (deflated 41%)
2022-12-01T11:16:35.3930501Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110236.xml (deflated 41%)
2022-12-01T11:16:35.3930955Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110245.xml (deflated 42%)
2022-12-01T11:16:35.3931384Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110255.xml (deflated 41%)
2022-12-01T11:16:35.3931814Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110257.xml (deflated 41%)
2022-12-01T11:16:35.3932245Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110259.xml (deflated 42%)
2022-12-01T11:16:35.3932676Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110301.xml (deflated 42%)
2022-12-01T11:16:35.3933116Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110303.xml (deflated 41%)
2022-12-01T11:16:35.3933616Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110305.xml (deflated 41%)
2022-12-01T11:16:35.3934054Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110307.xml (deflated 41%)
2022-12-01T11:16:35.3934487Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110309.xml (deflated 42%)
2022-12-01T11:16:35.3934920Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110312.xml (deflated 42%)
2022-12-01T11:16:35.3935338Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110314.xml (deflated 42%)
2022-12-01T11:16:35.3935773Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110316.xml (deflated 42%)
2022-12-01T11:16:35.3936213Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110318.xml (deflated 42%)
2022-12-01T11:16:35.3936646Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110320.xml (deflated 42%)
2022-12-01T11:16:35.3937077Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110322.xml (deflated 42%)
2022-12-01T11:16:35.3937511Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110324.xml (deflated 42%)
2022-12-01T11:16:35.3937951Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110326.xml (deflated 40%)
2022-12-01T11:16:35.3938385Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110332.xml (deflated 41%)
2022-12-01T11:16:35.3938824Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110338.xml (deflated 41%)
2022-12-01T11:16:35.3939259Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110341.xml (deflated 42%)
2022-12-01T11:16:35.3939674Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110343.xml (deflated 42%)
2022-12-01T11:16:35.3940105Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110347.xml (deflated 40%)
2022-12-01T11:16:35.3940539Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110355.xml (deflated 41%)
2022-12-01T11:16:35.3940971Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110404.xml (deflated 40%)
2022-12-01T11:16:35.3941462Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110414.xml (deflated 42%)
2022-12-01T11:16:35.3941910Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110417.xml (deflated 41%)
2022-12-01T11:16:35.3942343Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110421.xml (deflated 40%)
2022-12-01T11:16:35.3942775Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110428.xml (deflated 40%)
2022-12-01T11:16:35.3943206Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110434.xml (deflated 42%)
2022-12-01T11:16:35.3943636Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110438.xml (deflated 40%)
2022-12-01T11:16:35.3944074Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110447.xml (deflated 41%)
2022-12-01T11:16:35.3944557Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110457.xml (deflated 41%)
2022-12-01T11:16:35.3944991Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110506.xml (deflated 41%)
2022-12-01T11:16:35.3945419Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110516.xml (deflated 42%)
2022-12-01T11:16:35.3945853Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110522.xml (deflated 42%)
2022-12-01T11:16:35.3946281Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110529.xml (deflated 42%)
2022-12-01T11:16:35.3946719Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110535.xml (deflated 42%)
2022-12-01T11:16:35.3947152Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110541.xml (deflated 41%)
2022-12-01T11:16:35.3947578Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110550.xml (deflated 41%)
2022-12-01T11:16:35.3948012Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110600.xml (deflated 43%)
2022-12-01T11:16:35.3948444Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110602.xml (deflated 41%)
2022-12-01T11:16:35.3948863Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110611.xml (deflated 43%)
2022-12-01T11:16:35.3949298Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110613.xml (deflated 43%)
2022-12-01T11:16:35.3950131Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110615.xml (deflated 41%)
2022-12-01T11:16:35.3950573Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110624.xml (deflated 42%)
2022-12-01T11:16:35.3951003Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110626.xml (deflated 42%)
2022-12-01T11:16:35.3951434Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110628.xml (deflated 41%)
2022-12-01T11:16:35.3951870Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110637.xml (deflated 41%)
2022-12-01T11:16:35.3952305Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110639.xml (deflated 41%)
2022-12-01T11:16:35.3952813Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110641.xml (deflated 41%)
2022-12-01T11:16:35.3953260Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110644.xml (deflated 42%)
2022-12-01T11:16:35.3953694Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110646.xml (deflated 41%)
2022-12-01T11:16:35.3954109Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110648.xml (deflated 41%)
2022-12-01T11:16:35.3954541Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110650.xml (deflated 41%)
2022-12-01T11:16:35.3954973Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110652.xml (deflated 41%)
2022-12-01T11:16:35.3955409Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110654.xml (deflated 41%)
2022-12-01T11:16:35.3955932Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110702.xml (deflated 42%)
2022-12-01T11:16:35.3956366Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110704.xml (deflated 42%)
2022-12-01T11:16:35.3956794Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110706.xml (deflated 42%)
2022-12-01T11:16:35.3957228Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110708.xml (deflated 41%)
2022-12-01T11:16:35.3957663Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110718.xml (deflated 41%)
2022-12-01T11:16:35.3958100Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110720.xml (deflated 41%)
2022-12-01T11:16:35.3958523Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110722.xml (deflated 41%)
2022-12-01T11:16:35.3958968Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110724.xml (deflated 41%)
2022-12-01T11:16:35.3959403Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110734.xml (deflated 41%)
2022-12-01T11:16:35.3959840Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110742.xml (deflated 41%)
2022-12-01T11:16:35.3960274Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110751.xml (deflated 40%)
2022-12-01T11:16:35.3960713Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110801.xml (deflated 42%)
2022-12-01T11:16:35.3961150Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110803.xml (deflated 42%)
2022-12-01T11:16:35.3961584Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110805.xml (deflated 42%)
2022-12-01T11:16:35.3962017Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110814.xml (deflated 41%)
2022-12-01T11:16:35.3962451Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110822.xml (deflated 42%)
2022-12-01T11:16:35.3962880Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110824.xml (deflated 41%)
2022-12-01T11:16:35.3963301Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110833.xml (deflated 40%)
2022-12-01T11:16:35.3963790Z   adding: test/test-reports/dist-ucc/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110835.xml (deflated 41%)
2022-12-01T11:16:35.3964253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102113.xml (deflated 42%)
2022-12-01T11:16:35.3964925Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102122.xml (deflated 42%)
2022-12-01T11:16:35.3965368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102125.xml (deflated 42%)
2022-12-01T11:16:35.3965808Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102131.xml (deflated 43%)
2022-12-01T11:16:35.3966246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102135.xml (deflated 42%)
2022-12-01T11:16:35.3966754Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102137.xml (deflated 42%)
2022-12-01T11:16:35.3967200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102139.xml (deflated 40%)
2022-12-01T11:16:35.3967684Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102149.xml (deflated 40%)
2022-12-01T11:16:35.3968132Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102159.xml (deflated 40%)
2022-12-01T11:16:35.3968547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102208.xml (deflated 40%)
2022-12-01T11:16:35.3968985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102217.xml (deflated 40%)
2022-12-01T11:16:35.3969430Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102227.xml (deflated 40%)
2022-12-01T11:16:35.3969870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102237.xml (deflated 40%)
2022-12-01T11:16:35.3970364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102246.xml (deflated 43%)
2022-12-01T11:16:35.3970797Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102250.xml (deflated 42%)
2022-12-01T11:16:35.3971227Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102257.xml (deflated 42%)
2022-12-01T11:16:35.3971664Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102300.xml (deflated 42%)
2022-12-01T11:16:35.3972104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102309.xml (deflated 42%)
2022-12-01T11:16:35.3972547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102311.xml (deflated 46%)
2022-12-01T11:16:35.3972965Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102321.xml (deflated 47%)
2022-12-01T11:16:35.3973398Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102331.xml (deflated 48%)
2022-12-01T11:16:35.3973829Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102340.xml (deflated 46%)
2022-12-01T11:16:35.3974261Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102350.xml (deflated 41%)
2022-12-01T11:16:35.3974750Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102352.xml (deflated 44%)
2022-12-01T11:16:35.3975204Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102354.xml (deflated 43%)
2022-12-01T11:16:35.3975642Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102356.xml (deflated 43%)
2022-12-01T11:16:35.3976073Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102358.xml (deflated 44%)
2022-12-01T11:16:35.3976509Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102400.xml (deflated 43%)
2022-12-01T11:16:35.3976941Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102402.xml (deflated 41%)
2022-12-01T11:16:35.3977377Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102404.xml (deflated 40%)
2022-12-01T11:16:35.3977859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102414.xml (deflated 40%)
2022-12-01T11:16:35.3978292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102424.xml (deflated 41%)
2022-12-01T11:16:35.3978721Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102426.xml (deflated 41%)
2022-12-01T11:16:35.3979151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102428.xml (deflated 40%)
2022-12-01T11:16:35.3979584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102438.xml (deflated 40%)
2022-12-01T11:16:35.3980015Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102448.xml (deflated 42%)
2022-12-01T11:16:35.3980456Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102452.xml (deflated 42%)
2022-12-01T11:16:35.3980893Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102456.xml (deflated 40%)
2022-12-01T11:16:35.3981331Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102505.xml (deflated 41%)
2022-12-01T11:16:35.3981766Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102516.xml (deflated 40%)
2022-12-01T11:16:35.3982182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102525.xml (deflated 41%)
2022-12-01T11:16:35.3982619Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102527.xml (deflated 41%)
2022-12-01T11:16:35.3983064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102530.xml (deflated 41%)
2022-12-01T11:16:35.3983507Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102532.xml (deflated 41%)
2022-12-01T11:16:35.3983942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102534.xml (deflated 41%)
2022-12-01T11:16:35.3984376Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102536.xml (deflated 41%)
2022-12-01T11:16:35.3984812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102538.xml (deflated 41%)
2022-12-01T11:16:35.3985246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102540.xml (deflated 41%)
2022-12-01T11:16:35.3985737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102542.xml (deflated 41%)
2022-12-01T11:16:35.3986183Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102544.xml (deflated 42%)
2022-12-01T11:16:35.3986615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102546.xml (deflated 41%)
2022-12-01T11:16:35.3987029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102548.xml (deflated 42%)
2022-12-01T11:16:35.3987467Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102551.xml (deflated 41%)
2022-12-01T11:16:35.3987942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102553.xml (deflated 42%)
2022-12-01T11:16:35.3988380Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102555.xml (deflated 41%)
2022-12-01T11:16:35.3988888Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102557.xml (deflated 41%)
2022-12-01T11:16:35.3989323Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102559.xml (deflated 41%)
2022-12-01T11:16:35.3990106Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102601.xml (deflated 41%)
2022-12-01T11:16:35.3990543Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102603.xml (deflated 42%)
2022-12-01T11:16:35.3990982Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102605.xml (deflated 42%)
2022-12-01T11:16:35.3991418Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102607.xml (deflated 41%)
2022-12-01T11:16:35.3991839Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102609.xml (deflated 42%)
2022-12-01T11:16:35.3992273Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102611.xml (deflated 42%)
2022-12-01T11:16:35.3992706Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102613.xml (deflated 42%)
2022-12-01T11:16:35.3993139Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102615.xml (deflated 43%)
2022-12-01T11:16:35.3993574Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102617.xml (deflated 43%)
2022-12-01T11:16:35.3994013Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102620.xml (deflated 42%)
2022-12-01T11:16:35.3994450Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102622.xml (deflated 41%)
2022-12-01T11:16:35.3994883Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102631.xml (deflated 42%)
2022-12-01T11:16:35.3995310Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102633.xml (deflated 42%)
2022-12-01T11:16:35.3995740Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102635.xml (deflated 41%)
2022-12-01T11:16:35.3996177Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102637.xml (deflated 41%)
2022-12-01T11:16:35.3996842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102645.xml (deflated 40%)
2022-12-01T11:16:35.3997408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102654.xml (deflated 41%)
2022-12-01T11:16:35.3997866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102702.xml (deflated 42%)
2022-12-01T11:16:35.3998300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102704.xml (deflated 42%)
2022-12-01T11:16:35.3998731Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102706.xml (deflated 41%)
2022-12-01T11:16:35.3999167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102715.xml (deflated 41%)
2022-12-01T11:16:35.3999597Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102724.xml (deflated 43%)
2022-12-01T11:16:35.4000146Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102726.xml (deflated 41%)
2022-12-01T11:16:35.4000585Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102735.xml (deflated 42%)
2022-12-01T11:16:35.4001016Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102737.xml (deflated 43%)
2022-12-01T11:16:35.4001431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102744.xml (deflated 42%)
2022-12-01T11:16:35.4001865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102746.xml (deflated 42%)
2022-12-01T11:16:35.4002296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102748.xml (deflated 42%)
2022-12-01T11:16:35.4002735Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102752.xml (deflated 41%)
2022-12-01T11:16:35.4003176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102802.xml (deflated 42%)
2022-12-01T11:16:35.4003607Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102804.xml (deflated 42%)
2022-12-01T11:16:35.4004036Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102808.xml (deflated 42%)
2022-12-01T11:16:35.4004520Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102810.xml (deflated 42%)
2022-12-01T11:16:35.4004966Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102816.xml (deflated 43%)
2022-12-01T11:16:35.4005403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102818.xml (deflated 42%)
2022-12-01T11:16:35.4005824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102820.xml (deflated 41%)
2022-12-01T11:16:35.4006254Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102829.xml (deflated 41%)
2022-12-01T11:16:35.4006691Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102838.xml (deflated 42%)
2022-12-01T11:16:35.4007121Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102840.xml (deflated 40%)
2022-12-01T11:16:35.4007557Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102850.xml (deflated 42%)
2022-12-01T11:16:35.4008042Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102852.xml (deflated 42%)
2022-12-01T11:16:35.4008495Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102858.xml (deflated 41%)
2022-12-01T11:16:35.4008928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102907.xml (deflated 42%)
2022-12-01T11:16:35.4009356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102913.xml (deflated 42%)
2022-12-01T11:16:35.4009786Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102915.xml (deflated 42%)
2022-12-01T11:16:35.4010219Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102917.xml (deflated 41%)
2022-12-01T11:16:35.4010640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102927.xml (deflated 42%)
2022-12-01T11:16:35.4011141Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102929.xml (deflated 42%)
2022-12-01T11:16:35.4011573Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102935.xml (deflated 42%)
2022-12-01T11:16:35.4012001Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102937.xml (deflated 42%)
2022-12-01T11:16:35.4012431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102944.xml (deflated 42%)
2022-12-01T11:16:35.4012869Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102946.xml (deflated 43%)
2022-12-01T11:16:35.4013304Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102948.xml (deflated 43%)
2022-12-01T11:16:35.4013743Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102950.xml (deflated 41%)
2022-12-01T11:16:35.4014176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102952.xml (deflated 41%)
2022-12-01T11:16:35.4014612Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201102954.xml (deflated 41%)
2022-12-01T11:16:35.4015028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103001.xml (deflated 41%)
2022-12-01T11:16:35.4015462Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103009.xml (deflated 42%)
2022-12-01T11:16:35.4015894Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103015.xml (deflated 41%)
2022-12-01T11:16:35.4016335Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103023.xml (deflated 41%)
2022-12-01T11:16:35.4016771Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103029.xml (deflated 40%)
2022-12-01T11:16:35.4017203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103037.xml (deflated 41%)
2022-12-01T11:16:35.4017634Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103045.xml (deflated 41%)
2022-12-01T11:16:35.4018070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103052.xml (deflated 42%)
2022-12-01T11:16:35.4018501Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103053.xml (deflated 42%)
2022-12-01T11:16:35.4018981Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103057.xml (deflated 42%)
2022-12-01T11:16:35.4019429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103059.xml (deflated 42%)
2022-12-01T11:16:35.4019845Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103102.xml (deflated 44%)
2022-12-01T11:16:35.4020276Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103104.xml (deflated 42%)
2022-12-01T11:16:35.4020708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103108.xml (deflated 42%)
2022-12-01T11:16:35.4021141Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103112.xml (deflated 42%)
2022-12-01T11:16:35.4021572Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103116.xml (deflated 40%)
2022-12-01T11:16:35.4022069Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103124.xml (deflated 40%)
2022-12-01T11:16:35.4022504Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103132.xml (deflated 42%)
2022-12-01T11:16:35.4022932Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103136.xml (deflated 42%)
2022-12-01T11:16:35.4023364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103140.xml (deflated 40%)
2022-12-01T11:16:35.4023798Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103149.xml (deflated 40%)
2022-12-01T11:16:35.4024218Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103158.xml (deflated 40%)
2022-12-01T11:16:35.4024660Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103208.xml (deflated 40%)
2022-12-01T11:16:35.4025093Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103217.xml (deflated 42%)
2022-12-01T11:16:35.4025523Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103221.xml (deflated 43%)
2022-12-01T11:16:35.4025956Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103223.xml (deflated 42%)
2022-12-01T11:16:35.4026390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103227.xml (deflated 41%)
2022-12-01T11:16:35.4026825Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103237.xml (deflated 42%)
2022-12-01T11:16:35.4027261Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103240.xml (deflated 42%)
2022-12-01T11:16:35.4027695Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103244.xml (deflated 40%)
2022-12-01T11:16:35.4028131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103254.xml (deflated 40%)
2022-12-01T11:16:35.4028592Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103305.xml (deflated 42%)
2022-12-01T11:16:35.4029012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103309.xml (deflated 41%)
2022-12-01T11:16:35.4029804Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103318.xml (deflated 42%)
2022-12-01T11:16:35.4030325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103320.xml (deflated 42%)
2022-12-01T11:16:35.4030773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103322.xml (deflated 42%)
2022-12-01T11:16:35.4031207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103325.xml (deflated 41%)
2022-12-01T11:16:35.4031638Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103327.xml (deflated 41%)
2022-12-01T11:16:35.4032070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103329.xml (deflated 41%)
2022-12-01T11:16:35.4032500Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103331.xml (deflated 41%)
2022-12-01T11:16:35.4033012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103333.xml (deflated 41%)
2022-12-01T11:16:35.4033443Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103335.xml (deflated 41%)
2022-12-01T11:16:35.4033859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103337.xml (deflated 41%)
2022-12-01T11:16:35.4034290Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103339.xml (deflated 42%)
2022-12-01T11:16:35.4034722Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103341.xml (deflated 42%)
2022-12-01T11:16:35.4035158Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103343.xml (deflated 42%)
2022-12-01T11:16:35.4035594Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103347.xml (deflated 40%)
2022-12-01T11:16:35.4036031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103356.xml (deflated 40%)
2022-12-01T11:16:35.4036460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103406.xml (deflated 41%)
2022-12-01T11:16:35.4036894Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103408.xml (deflated 40%)
2022-12-01T11:16:35.4037325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103418.xml (deflated 40%)
2022-12-01T11:16:35.4037755Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103426.xml (deflated 40%)
2022-12-01T11:16:35.4038176Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103444.xml (deflated 41%)
2022-12-01T11:16:35.4038610Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103454.xml (deflated 41%)
2022-12-01T11:16:35.4039039Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103505.xml (deflated 41%)
2022-12-01T11:16:35.4039471Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103514.xml (deflated 41%)
2022-12-01T11:16:35.4039902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103524.xml (deflated 42%)
2022-12-01T11:16:35.4040336Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103528.xml (deflated 42%)
2022-12-01T11:16:35.4040815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103532.xml (deflated 42%)
2022-12-01T11:16:35.4041268Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103541.xml (deflated 40%)
2022-12-01T11:16:35.4041697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103550.xml (deflated 42%)
2022-12-01T11:16:35.4042129Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103554.xml (deflated 40%)
2022-12-01T11:16:35.4042560Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103603.xml (deflated 42%)
2022-12-01T11:16:35.4042975Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103606.xml (deflated 41%)
2022-12-01T11:16:35.4043412Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103616.xml (deflated 41%)
2022-12-01T11:16:35.4043905Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103625.xml (deflated 41%)
2022-12-01T11:16:35.4044340Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103634.xml (deflated 42%)
2022-12-01T11:16:35.4044770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103638.xml (deflated 42%)
2022-12-01T11:16:35.4045202Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103642.xml (deflated 42%)
2022-12-01T11:16:35.4045634Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103646.xml (deflated 41%)
2022-12-01T11:16:35.4046069Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103655.xml (deflated 40%)
2022-12-01T11:16:35.4046502Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103702.xml (deflated 41%)
2022-12-01T11:16:35.4046933Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103708.xml (deflated 42%)
2022-12-01T11:16:35.4047347Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103712.xml (deflated 42%)
2022-12-01T11:16:35.4047779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103716.xml (deflated 40%)
2022-12-01T11:16:35.4048213Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103723.xml (deflated 41%)
2022-12-01T11:16:35.4048645Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103725.xml (deflated 41%)
2022-12-01T11:16:35.4049086Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103727.xml (deflated 41%)
2022-12-01T11:16:35.4049517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103737.xml (deflated 41%)
2022-12-01T11:16:35.4049946Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103739.xml (deflated 41%)
2022-12-01T11:16:35.4050378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103741.xml (deflated 41%)
2022-12-01T11:16:35.4050813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103751.xml (deflated 42%)
2022-12-01T11:16:35.4051249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103755.xml (deflated 41%)
2022-12-01T11:16:35.4051731Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103801.xml (deflated 40%)
2022-12-01T11:16:35.4052166Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103810.xml (deflated 41%)
2022-12-01T11:16:35.4052600Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103816.xml (deflated 41%)
2022-12-01T11:16:35.4053025Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103823.xml (deflated 40%)
2022-12-01T11:16:35.4053455Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103829.xml (deflated 40%)
2022-12-01T11:16:35.4053889Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103838.xml (deflated 43%)
2022-12-01T11:16:35.4054326Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103840.xml (deflated 43%)
2022-12-01T11:16:35.4054824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103842.xml (deflated 42%)
2022-12-01T11:16:35.4055256Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103844.xml (deflated 42%)
2022-12-01T11:16:35.4055687Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103846.xml (deflated 41%)
2022-12-01T11:16:35.4056119Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103911.xml (deflated 41%)
2022-12-01T11:16:35.4056536Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103936.xml (deflated 42%)
2022-12-01T11:16:35.4056979Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103938.xml (deflated 41%)
2022-12-01T11:16:35.4057421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103940.xml (deflated 41%)
2022-12-01T11:16:35.4057854Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103942.xml (deflated 41%)
2022-12-01T11:16:35.4058287Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103944.xml (deflated 41%)
2022-12-01T11:16:35.4058722Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103946.xml (deflated 41%)
2022-12-01T11:16:35.4059155Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201103954.xml (deflated 41%)
2022-12-01T11:16:35.4059590Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104002.xml (deflated 41%)
2022-12-01T11:16:35.4060033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104011.xml (deflated 41%)
2022-12-01T11:16:35.4060473Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104019.xml (deflated 42%)
2022-12-01T11:16:35.4060906Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104023.xml (deflated 42%)
2022-12-01T11:16:35.4061324Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104025.xml (deflated 42%)
2022-12-01T11:16:35.4061757Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104027.xml (deflated 42%)
2022-12-01T11:16:35.4062185Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104029.xml (deflated 40%)
2022-12-01T11:16:35.4062666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104035.xml (deflated 40%)
2022-12-01T11:16:35.4063110Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104042.xml (deflated 41%)
2022-12-01T11:16:35.4063539Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104044.xml (deflated 42%)
2022-12-01T11:16:35.4063964Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104046.xml (deflated 42%)
2022-12-01T11:16:35.4064390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104050.xml (deflated 41%)
2022-12-01T11:16:35.4064816Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104100.xml (deflated 41%)
2022-12-01T11:16:35.4065329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104109.xml (deflated 40%)
2022-12-01T11:16:35.4065745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104118.xml (deflated 42%)
2022-12-01T11:16:35.4066173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104122.xml (deflated 41%)
2022-12-01T11:16:35.4066598Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104126.xml (deflated 40%)
2022-12-01T11:16:35.4067028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104132.xml (deflated 40%)
2022-12-01T11:16:35.4067460Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104138.xml (deflated 42%)
2022-12-01T11:16:35.4067940Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104141.xml (deflated 41%)
2022-12-01T11:16:35.4068381Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104143.xml (deflated 42%)
2022-12-01T11:16:35.4068812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104145.xml (deflated 42%)
2022-12-01T11:16:35.4069245Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104147.xml (deflated 42%)
2022-12-01T11:16:35.4070045Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104149.xml (deflated 41%)
2022-12-01T11:16:35.4070469Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104151.xml (deflated 41%)
2022-12-01T11:16:35.4070934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104153.xml (deflated 42%)
2022-12-01T11:16:35.4071370Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104155.xml (deflated 41%)
2022-12-01T11:16:35.4071805Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104157.xml (deflated 41%)
2022-12-01T11:16:35.4072233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104159.xml (deflated 42%)
2022-12-01T11:16:35.4072663Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104201.xml (deflated 41%)
2022-12-01T11:16:35.4073095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104213.xml (deflated 42%)
2022-12-01T11:16:35.4073595Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104215.xml (deflated 40%)
2022-12-01T11:16:35.4074050Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104225.xml (deflated 41%)
2022-12-01T11:16:35.4074484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104234.xml (deflated 41%)
2022-12-01T11:16:35.4074917Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104236.xml (deflated 41%)
2022-12-01T11:16:35.4075332Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104244.xml (deflated 41%)
2022-12-01T11:16:35.4075769Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104252.xml (deflated 42%)
2022-12-01T11:16:35.4076200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104254.xml (deflated 42%)
2022-12-01T11:16:35.4076717Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104256.xml (deflated 42%)
2022-12-01T11:16:35.4077154Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104258.xml (deflated 42%)
2022-12-01T11:16:35.4077584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104300.xml (deflated 41%)
2022-12-01T11:16:35.4078017Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104310.xml (deflated 40%)
2022-12-01T11:16:35.4078451Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104320.xml (deflated 42%)
2022-12-01T11:16:35.4078886Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104322.xml (deflated 42%)
2022-12-01T11:16:35.4079334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104324.xml (deflated 41%)
2022-12-01T11:16:35.4079752Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104326.xml (deflated 43%)
2022-12-01T11:16:35.4080187Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104328.xml (deflated 42%)
2022-12-01T11:16:35.4080618Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104330.xml (deflated 42%)
2022-12-01T11:16:35.4081055Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104332.xml (deflated 42%)
2022-12-01T11:16:35.4081487Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104334.xml (deflated 43%)
2022-12-01T11:16:35.4081927Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104336.xml (deflated 41%)
2022-12-01T11:16:35.4082360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104344.xml (deflated 40%)
2022-12-01T11:16:35.4082793Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104352.xml (deflated 41%)
2022-12-01T11:16:35.4083225Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104400.xml (deflated 43%)
2022-12-01T11:16:35.4083656Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104402.xml (deflated 43%)
2022-12-01T11:16:35.4084088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104404.xml (deflated 43%)
2022-12-01T11:16:35.4084551Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104406.xml (deflated 43%)
2022-12-01T11:16:35.4084998Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104408.xml (deflated 42%)
2022-12-01T11:16:35.4085433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104410.xml (deflated 42%)
2022-12-01T11:16:35.4085866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104412.xml (deflated 42%)
2022-12-01T11:16:35.4086296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104421.xml (deflated 41%)
2022-12-01T11:16:35.4086731Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104423.xml (deflated 40%)
2022-12-01T11:16:35.4087170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104432.xml (deflated 41%)
2022-12-01T11:16:35.4087663Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104441.xml (deflated 40%)
2022-12-01T11:16:35.4088093Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104454.xml (deflated 40%)
2022-12-01T11:16:35.4088524Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104823.xml (deflated 41%)
2022-12-01T11:16:35.4088937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104832.xml (deflated 42%)
2022-12-01T11:16:35.4089368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104834.xml (deflated 42%)
2022-12-01T11:16:35.4089808Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104841.xml (deflated 43%)
2022-12-01T11:16:35.4090242Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104845.xml (deflated 42%)
2022-12-01T11:16:35.4090678Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104847.xml (deflated 42%)
2022-12-01T11:16:35.4091112Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104849.xml (deflated 40%)
2022-12-01T11:16:35.4091546Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104859.xml (deflated 40%)
2022-12-01T11:16:35.4092005Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104908.xml (deflated 40%)
2022-12-01T11:16:35.4092445Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104917.xml (deflated 39%)
2022-12-01T11:16:35.4092884Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104927.xml (deflated 39%)
2022-12-01T11:16:35.4093298Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104937.xml (deflated 40%)
2022-12-01T11:16:35.4093735Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104947.xml (deflated 40%)
2022-12-01T11:16:35.4094165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201104956.xml (deflated 42%)
2022-12-01T11:16:35.4094597Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105000.xml (deflated 42%)
2022-12-01T11:16:35.4095031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105006.xml (deflated 42%)
2022-12-01T11:16:35.4095509Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105010.xml (deflated 42%)
2022-12-01T11:16:35.4095951Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105019.xml (deflated 42%)
2022-12-01T11:16:35.4096383Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105021.xml (deflated 46%)
2022-12-01T11:16:35.4096813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105031.xml (deflated 47%)
2022-12-01T11:16:35.4097241Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105040.xml (deflated 48%)
2022-12-01T11:16:35.4097671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105050.xml (deflated 46%)
2022-12-01T11:16:35.4098152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105059.xml (deflated 41%)
2022-12-01T11:16:35.4098581Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105101.xml (deflated 44%)
2022-12-01T11:16:35.4099012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105104.xml (deflated 43%)
2022-12-01T11:16:35.4099445Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105106.xml (deflated 43%)
2022-12-01T11:16:35.4099882Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105108.xml (deflated 44%)
2022-12-01T11:16:35.4100313Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105110.xml (deflated 43%)
2022-12-01T11:16:35.4100752Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105112.xml (deflated 42%)
2022-12-01T11:16:35.4101190Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105114.xml (deflated 40%)
2022-12-01T11:16:35.4101619Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105124.xml (deflated 40%)
2022-12-01T11:16:35.4102047Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105134.xml (deflated 41%)
2022-12-01T11:16:35.4102461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105136.xml (deflated 41%)
2022-12-01T11:16:35.4102895Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105138.xml (deflated 40%)
2022-12-01T11:16:35.4103332Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105148.xml (deflated 40%)
2022-12-01T11:16:35.4103770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105158.xml (deflated 42%)
2022-12-01T11:16:35.4104200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105202.xml (deflated 42%)
2022-12-01T11:16:35.4104632Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105206.xml (deflated 40%)
2022-12-01T11:16:35.4105059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105215.xml (deflated 40%)
2022-12-01T11:16:35.4105494Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105226.xml (deflated 40%)
2022-12-01T11:16:35.4105972Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105235.xml (deflated 41%)
2022-12-01T11:16:35.4106422Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105237.xml (deflated 41%)
2022-12-01T11:16:35.4106848Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105239.xml (deflated 41%)
2022-12-01T11:16:35.4107266Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105241.xml (deflated 41%)
2022-12-01T11:16:35.4107697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105243.xml (deflated 41%)
2022-12-01T11:16:35.4108132Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105246.xml (deflated 41%)
2022-12-01T11:16:35.4108562Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105248.xml (deflated 41%)
2022-12-01T11:16:35.4109058Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105250.xml (deflated 41%)
2022-12-01T11:16:35.4109852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105252.xml (deflated 41%)
2022-12-01T11:16:35.4110290Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105254.xml (deflated 42%)
2022-12-01T11:16:35.4110720Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105256.xml (deflated 41%)
2022-12-01T11:16:35.4111145Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105258.xml (deflated 42%)
2022-12-01T11:16:35.4111580Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105300.xml (deflated 41%)
2022-12-01T11:16:35.4112005Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105302.xml (deflated 42%)
2022-12-01T11:16:35.4112440Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105304.xml (deflated 41%)
2022-12-01T11:16:35.4112901Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105306.xml (deflated 41%)
2022-12-01T11:16:35.4113329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105309.xml (deflated 41%)
2022-12-01T11:16:35.4113753Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105311.xml (deflated 41%)
2022-12-01T11:16:35.4114187Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105313.xml (deflated 41%)
2022-12-01T11:16:35.4114625Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105315.xml (deflated 41%)
2022-12-01T11:16:35.4115059Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105317.xml (deflated 41%)
2022-12-01T11:16:35.4115489Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105319.xml (deflated 42%)
2022-12-01T11:16:35.4115921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105321.xml (deflated 42%)
2022-12-01T11:16:35.4116355Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105323.xml (deflated 42%)
2022-12-01T11:16:35.4116770Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105325.xml (deflated 43%)
2022-12-01T11:16:35.4117274Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105327.xml (deflated 43%)
2022-12-01T11:16:35.4117731Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105330.xml (deflated 41%)
2022-12-01T11:16:35.4118163Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105332.xml (deflated 41%)
2022-12-01T11:16:35.4118591Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105341.xml (deflated 41%)
2022-12-01T11:16:35.4119024Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105343.xml (deflated 42%)
2022-12-01T11:16:35.4119450Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105345.xml (deflated 41%)
2022-12-01T11:16:35.4119906Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105347.xml (deflated 40%)
2022-12-01T11:16:35.4120421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105356.xml (deflated 40%)
2022-12-01T11:16:35.4120851Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105404.xml (deflated 41%)
2022-12-01T11:16:35.4121265Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105412.xml (deflated 42%)
2022-12-01T11:16:35.4121696Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105414.xml (deflated 42%)
2022-12-01T11:16:35.4122127Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105417.xml (deflated 41%)
2022-12-01T11:16:35.4122563Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105426.xml (deflated 40%)
2022-12-01T11:16:35.4122997Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105434.xml (deflated 42%)
2022-12-01T11:16:35.4123431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105437.xml (deflated 41%)
2022-12-01T11:16:35.4123863Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105446.xml (deflated 42%)
2022-12-01T11:16:35.4124297Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105448.xml (deflated 42%)
2022-12-01T11:16:35.4124731Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105454.xml (deflated 42%)
2022-12-01T11:16:35.4125165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105456.xml (deflated 42%)
2022-12-01T11:16:35.4125587Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105458.xml (deflated 42%)
2022-12-01T11:16:35.4126025Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105502.xml (deflated 41%)
2022-12-01T11:16:35.4126464Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105512.xml (deflated 42%)
2022-12-01T11:16:35.4126902Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105514.xml (deflated 42%)
2022-12-01T11:16:35.4127338Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105518.xml (deflated 42%)
2022-12-01T11:16:35.4127771Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105520.xml (deflated 42%)
2022-12-01T11:16:35.4128256Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105526.xml (deflated 43%)
2022-12-01T11:16:35.4128706Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105528.xml (deflated 42%)
2022-12-01T11:16:35.4129141Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105530.xml (deflated 41%)
2022-12-01T11:16:35.4129572Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105539.xml (deflated 40%)
2022-12-01T11:16:35.4130004Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105548.xml (deflated 42%)
2022-12-01T11:16:35.4130421Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105550.xml (deflated 40%)
2022-12-01T11:16:35.4130923Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105600.xml (deflated 42%)
2022-12-01T11:16:35.4131356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105602.xml (deflated 42%)
2022-12-01T11:16:35.4131791Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105608.xml (deflated 41%)
2022-12-01T11:16:35.4132223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105617.xml (deflated 42%)
2022-12-01T11:16:35.4132651Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105623.xml (deflated 42%)
2022-12-01T11:16:35.4133080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105625.xml (deflated 42%)
2022-12-01T11:16:35.4133515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105627.xml (deflated 40%)
2022-12-01T11:16:35.4133947Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105637.xml (deflated 42%)
2022-12-01T11:16:35.4134378Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105639.xml (deflated 42%)
2022-12-01T11:16:35.4134794Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105645.xml (deflated 42%)
2022-12-01T11:16:35.4135222Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105647.xml (deflated 42%)
2022-12-01T11:16:35.4135654Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105654.xml (deflated 42%)
2022-12-01T11:16:35.4136085Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105656.xml (deflated 43%)
2022-12-01T11:16:35.4136516Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105658.xml (deflated 42%)
2022-12-01T11:16:35.4136942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105700.xml (deflated 41%)
2022-12-01T11:16:35.4137393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105702.xml (deflated 41%)
2022-12-01T11:16:35.4137819Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105704.xml (deflated 41%)
2022-12-01T11:16:35.4138244Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105710.xml (deflated 41%)
2022-12-01T11:16:35.4138716Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105718.xml (deflated 41%)
2022-12-01T11:16:35.4139167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105724.xml (deflated 41%)
2022-12-01T11:16:35.4139584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105732.xml (deflated 40%)
2022-12-01T11:16:35.4140013Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105738.xml (deflated 40%)
2022-12-01T11:16:35.4140444Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105746.xml (deflated 41%)
2022-12-01T11:16:35.4140872Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105754.xml (deflated 41%)
2022-12-01T11:16:35.4141303Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105801.xml (deflated 42%)
2022-12-01T11:16:35.4141809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105803.xml (deflated 42%)
2022-12-01T11:16:35.4142234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105807.xml (deflated 42%)
2022-12-01T11:16:35.4142663Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105809.xml (deflated 42%)
2022-12-01T11:16:35.4143087Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105811.xml (deflated 43%)
2022-12-01T11:16:35.4143516Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105813.xml (deflated 42%)
2022-12-01T11:16:35.4143936Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105817.xml (deflated 42%)
2022-12-01T11:16:35.4144374Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105821.xml (deflated 42%)
2022-12-01T11:16:35.4144801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105825.xml (deflated 41%)
2022-12-01T11:16:35.4145231Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105833.xml (deflated 40%)
2022-12-01T11:16:35.4145658Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105841.xml (deflated 42%)
2022-12-01T11:16:35.4146089Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105845.xml (deflated 42%)
2022-12-01T11:16:35.4146518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105849.xml (deflated 40%)
2022-12-01T11:16:35.4146954Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105858.xml (deflated 40%)
2022-12-01T11:16:35.4147389Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105907.xml (deflated 40%)
2022-12-01T11:16:35.4147811Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105916.xml (deflated 40%)
2022-12-01T11:16:35.4148227Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105925.xml (deflated 42%)
2022-12-01T11:16:35.4148655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105929.xml (deflated 43%)
2022-12-01T11:16:35.4149088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105931.xml (deflated 42%)
2022-12-01T11:16:35.4149953Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105935.xml (deflated 41%)
2022-12-01T11:16:35.4150420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105944.xml (deflated 42%)
2022-12-01T11:16:35.4150846Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105948.xml (deflated 42%)
2022-12-01T11:16:35.4151277Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201105952.xml (deflated 40%)
2022-12-01T11:16:35.4151708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110002.xml (deflated 40%)
2022-12-01T11:16:35.4152140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110013.xml (deflated 42%)
2022-12-01T11:16:35.4152579Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110016.xml (deflated 41%)
2022-12-01T11:16:35.4153084Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110026.xml (deflated 42%)
2022-12-01T11:16:35.4153500Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110028.xml (deflated 42%)
2022-12-01T11:16:35.4153930Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110030.xml (deflated 42%)
2022-12-01T11:16:35.4154359Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110033.xml (deflated 41%)
2022-12-01T11:16:35.4154784Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110035.xml (deflated 41%)
2022-12-01T11:16:35.4155216Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110037.xml (deflated 41%)
2022-12-01T11:16:35.4155648Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110039.xml (deflated 41%)
2022-12-01T11:16:35.4156081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110041.xml (deflated 41%)
2022-12-01T11:16:35.4156510Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110043.xml (deflated 41%)
2022-12-01T11:16:35.4156943Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110045.xml (deflated 42%)
2022-12-01T11:16:35.4157374Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110047.xml (deflated 42%)
2022-12-01T11:16:35.4157794Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110049.xml (deflated 42%)
2022-12-01T11:16:35.4158231Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110051.xml (deflated 42%)
2022-12-01T11:16:35.4158661Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110055.xml (deflated 41%)
2022-12-01T11:16:35.4159094Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110104.xml (deflated 41%)
2022-12-01T11:16:35.4159529Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110114.xml (deflated 41%)
2022-12-01T11:16:35.4159960Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110116.xml (deflated 41%)
2022-12-01T11:16:35.4160389Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110126.xml (deflated 40%)
2022-12-01T11:16:35.4160870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110134.xml (deflated 41%)
2022-12-01T11:16:35.4161319Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110152.xml (deflated 41%)
2022-12-01T11:16:35.4161754Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110202.xml (deflated 41%)
2022-12-01T11:16:35.4162183Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110212.xml (deflated 42%)
2022-12-01T11:16:35.4162601Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110222.xml (deflated 41%)
2022-12-01T11:16:35.4163026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110232.xml (deflated 42%)
2022-12-01T11:16:35.4163523Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110236.xml (deflated 42%)
2022-12-01T11:16:35.4163957Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110240.xml (deflated 43%)
2022-12-01T11:16:35.4164392Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110249.xml (deflated 41%)
2022-12-01T11:16:35.4164824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110258.xml (deflated 42%)
2022-12-01T11:16:35.4165260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110302.xml (deflated 41%)
2022-12-01T11:16:35.4165695Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110310.xml (deflated 42%)
2022-12-01T11:16:35.4166131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110314.xml (deflated 41%)
2022-12-01T11:16:35.4166565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110323.xml (deflated 41%)
2022-12-01T11:16:35.4166983Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110332.xml (deflated 41%)
2022-12-01T11:16:35.4167412Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110341.xml (deflated 42%)
2022-12-01T11:16:35.4167909Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110345.xml (deflated 42%)
2022-12-01T11:16:35.4168344Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110349.xml (deflated 42%)
2022-12-01T11:16:35.4168775Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110353.xml (deflated 41%)
2022-12-01T11:16:35.4169215Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110403.xml (deflated 40%)
2022-12-01T11:16:35.4169640Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110409.xml (deflated 41%)
2022-12-01T11:16:35.4170072Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110415.xml (deflated 42%)
2022-12-01T11:16:35.4170502Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110419.xml (deflated 42%)
2022-12-01T11:16:35.4170939Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110423.xml (deflated 40%)
2022-12-01T11:16:35.4171433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110429.xml (deflated 41%)
2022-12-01T11:16:35.4171869Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110431.xml (deflated 42%)
2022-12-01T11:16:35.4172302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110434.xml (deflated 41%)
2022-12-01T11:16:35.4172736Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110444.xml (deflated 41%)
2022-12-01T11:16:35.4173167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110446.xml (deflated 41%)
2022-12-01T11:16:35.4173596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110448.xml (deflated 40%)
2022-12-01T11:16:35.4174029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110457.xml (deflated 42%)
2022-12-01T11:16:35.4174525Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110501.xml (deflated 42%)
2022-12-01T11:16:35.4174953Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110507.xml (deflated 40%)
2022-12-01T11:16:35.4175381Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110516.xml (deflated 41%)
2022-12-01T11:16:35.4175807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110523.xml (deflated 40%)
2022-12-01T11:16:35.4176223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110529.xml (deflated 40%)
2022-12-01T11:16:35.4176648Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110535.xml (deflated 40%)
2022-12-01T11:16:35.4177085Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110544.xml (deflated 43%)
2022-12-01T11:16:35.4177517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110546.xml (deflated 43%)
2022-12-01T11:16:35.4177948Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110548.xml (deflated 42%)
2022-12-01T11:16:35.4178386Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110550.xml (deflated 42%)
2022-12-01T11:16:35.4178821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110552.xml (deflated 41%)
2022-12-01T11:16:35.4179249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110617.xml (deflated 41%)
2022-12-01T11:16:35.4179690Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110642.xml (deflated 42%)
2022-12-01T11:16:35.4180122Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110644.xml (deflated 42%)
2022-12-01T11:16:35.4180541Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110646.xml (deflated 41%)
2022-12-01T11:16:35.4180977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110649.xml (deflated 41%)
2022-12-01T11:16:35.4181410Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110651.xml (deflated 41%)
2022-12-01T11:16:35.4181844Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110653.xml (deflated 41%)
2022-12-01T11:16:35.4182325Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110701.xml (deflated 42%)
2022-12-01T11:16:35.4182773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110708.xml (deflated 41%)
2022-12-01T11:16:35.4183203Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110717.xml (deflated 41%)
2022-12-01T11:16:35.4183634Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110725.xml (deflated 42%)
2022-12-01T11:16:35.4184068Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110729.xml (deflated 42%)
2022-12-01T11:16:35.4184503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110731.xml (deflated 42%)
2022-12-01T11:16:35.4184934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110733.xml (deflated 41%)
2022-12-01T11:16:35.4185413Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110735.xml (deflated 40%)
2022-12-01T11:16:35.4185842Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110741.xml (deflated 40%)
2022-12-01T11:16:35.4186267Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110748.xml (deflated 41%)
2022-12-01T11:16:35.4186703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110750.xml (deflated 42%)
2022-12-01T11:16:35.4187138Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110752.xml (deflated 42%)
2022-12-01T11:16:35.4187577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110756.xml (deflated 40%)
2022-12-01T11:16:35.4188014Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110806.xml (deflated 41%)
2022-12-01T11:16:35.4188447Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110815.xml (deflated 41%)
2022-12-01T11:16:35.4188872Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110824.xml (deflated 42%)
2022-12-01T11:16:35.4189302Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110828.xml (deflated 41%)
2022-12-01T11:16:35.4190103Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110832.xml (deflated 40%)
2022-12-01T11:16:35.4190541Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110838.xml (deflated 40%)
2022-12-01T11:16:35.4190976Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110844.xml (deflated 42%)
2022-12-01T11:16:35.4191407Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110848.xml (deflated 42%)
2022-12-01T11:16:35.4191833Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110850.xml (deflated 42%)
2022-12-01T11:16:35.4192279Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110852.xml (deflated 42%)
2022-12-01T11:16:35.4192729Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110854.xml (deflated 42%)
2022-12-01T11:16:35.4193167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110855.xml (deflated 41%)
2022-12-01T11:16:35.4193683Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110857.xml (deflated 41%)
2022-12-01T11:16:35.4194162Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110859.xml (deflated 42%)
2022-12-01T11:16:35.4194602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110901.xml (deflated 41%)
2022-12-01T11:16:35.4195022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110903.xml (deflated 42%)
2022-12-01T11:16:35.4195461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110905.xml (deflated 42%)
2022-12-01T11:16:35.4195898Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110907.xml (deflated 41%)
2022-12-01T11:16:35.4196419Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110919.xml (deflated 42%)
2022-12-01T11:16:35.4196859Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110921.xml (deflated 41%)
2022-12-01T11:16:35.4197296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110931.xml (deflated 40%)
2022-12-01T11:16:35.4197762Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110940.xml (deflated 41%)
2022-12-01T11:16:35.4198200Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110942.xml (deflated 41%)
2022-12-01T11:16:35.4198633Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110950.xml (deflated 41%)
2022-12-01T11:16:35.4199070Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201110958.xml (deflated 42%)
2022-12-01T11:16:35.4199500Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111000.xml (deflated 42%)
2022-12-01T11:16:35.4199938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111002.xml (deflated 42%)
2022-12-01T11:16:35.4200353Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111004.xml (deflated 42%)
2022-12-01T11:16:35.4200789Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111006.xml (deflated 41%)
2022-12-01T11:16:35.4201220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111016.xml (deflated 41%)
2022-12-01T11:16:35.4201655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111026.xml (deflated 42%)
2022-12-01T11:16:35.4202091Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111028.xml (deflated 42%)
2022-12-01T11:16:35.4202515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111030.xml (deflated 41%)
2022-12-01T11:16:35.4202946Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111032.xml (deflated 44%)
2022-12-01T11:16:35.4203377Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111034.xml (deflated 42%)
2022-12-01T11:16:35.4203813Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111036.xml (deflated 42%)
2022-12-01T11:16:35.4204292Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111038.xml (deflated 42%)
2022-12-01T11:16:35.4204728Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111040.xml (deflated 43%)
2022-12-01T11:16:35.4205161Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111042.xml (deflated 41%)
2022-12-01T11:16:35.4205590Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111050.xml (deflated 41%)
2022-12-01T11:16:35.4206025Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111058.xml (deflated 41%)
2022-12-01T11:16:35.4206456Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111106.xml (deflated 43%)
2022-12-01T11:16:35.4206887Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111108.xml (deflated 43%)
2022-12-01T11:16:35.4207382Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111110.xml (deflated 43%)
2022-12-01T11:16:35.4207815Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111112.xml (deflated 44%)
2022-12-01T11:16:35.4208246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111114.xml (deflated 43%)
2022-12-01T11:16:35.4208674Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111116.xml (deflated 43%)
2022-12-01T11:16:35.4209104Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111118.xml (deflated 42%)
2022-12-01T11:16:35.4209522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111127.xml (deflated 42%)
2022-12-01T11:16:35.4209958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111129.xml (deflated 41%)
2022-12-01T11:16:35.4210393Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111138.xml (deflated 42%)
2022-12-01T11:16:35.4210824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111148.xml (deflated 41%)
2022-12-01T11:16:35.4211255Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20221201111201.xml (deflated 41%)
2022-12-01T11:16:35.4211588Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 71%)
2022-12-01T11:16:35.4211914Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%)
2022-12-01T11:16:35.4212238Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%)
2022-12-01T11:16:35.4212605Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 81%)
2022-12-01T11:16:35.4212947Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 79%)
2022-12-01T11:16:35.4213322Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 68%)
2022-12-01T11:16:35.4213629Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2022-12-01T11:16:35.4237661Z ##[group]Run # Remove any previous test reports if they exist
2022-12-01T11:16:35.4237850Z # Remove any previous test reports if they exist
2022-12-01T11:16:35.4237967Z rm -f usage-log-*.zip
2022-12-01T11:16:35.4238182Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2022-12-01T11:16:35.4238325Z # so check to see if the file exists first
2022-12-01T11:16:35.4238484Z if [ -f 'usage_log.txt' ]; then
2022-12-01T11:16:35.4238759Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2022-12-01T11:16:35.4238866Z fi
2022-12-01T11:16:35.4251163Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:35.4251249Z env:
2022-12-01T11:16:35.4251365Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:35.4251473Z   GPU_FLAG: --gpus all
2022-12-01T11:16:35.4251692Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304
2022-12-01T11:16:35.4251788Z ##[endgroup]
2022-12-01T11:16:35.5245437Z   adding: usage_log.txt (deflated 95%)
2022-12-01T11:16:35.5316250Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T11:16:35.5316350Z with:
2022-12-01T11:16:35.5316499Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T11:16:35.5316619Z   retention-days: 14
2022-12-01T11:16:35.5316740Z   if-no-files-found: warn
2022-12-01T11:16:35.5316859Z   path: test-jsons-*.zip
2022-12-01T11:16:35.5316960Z   name: artifact
2022-12-01T11:16:35.5317078Z   s3-bucket: gha-artifacts
2022-12-01T11:16:35.5317317Z   region: us-east-1
2022-12-01T11:16:35.5317392Z env:
2022-12-01T11:16:35.5317513Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:35.5317626Z   GPU_FLAG: --gpus all
2022-12-01T11:16:35.5317730Z ##[endgroup]
2022-12-01T11:16:35.9738284Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T11:16:35.9738968Z With the provided path, there will be 1 file uploaded
2022-12-01T11:16:35.9739349Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T11:16:35.9750781Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304.zip
2022-12-01T11:16:36.1082956Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304.zip
2022-12-01T11:16:36.1260537Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T11:16:36.1260837Z with:
2022-12-01T11:16:36.1261100Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T11:16:36.1261401Z   retention-days: 14
2022-12-01T11:16:36.1261672Z   if-no-files-found: error
2022-12-01T11:16:36.1261961Z   path: test-reports-*.zip
2022-12-01T11:16:36.1262225Z   name: artifact
2022-12-01T11:16:36.1262478Z   s3-bucket: gha-artifacts
2022-12-01T11:16:36.1262744Z   region: us-east-1
2022-12-01T11:16:36.1262959Z env:
2022-12-01T11:16:36.1263196Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:36.1263461Z   GPU_FLAG: --gpus all
2022-12-01T11:16:36.1263690Z ##[endgroup]
2022-12-01T11:16:36.5723508Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T11:16:36.5724266Z With the provided path, there will be 1 file uploaded
2022-12-01T11:16:36.5724649Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T11:16:36.5736352Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304.zip
2022-12-01T11:16:36.7933602Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304.zip
2022-12-01T11:16:36.8087080Z ##[group]Run seemethere/upload-artifact-s3@v5
2022-12-01T11:16:36.8087376Z with:
2022-12-01T11:16:36.8087665Z   s3-prefix: pytorch/pytorch/3591403534/1/artifact

2022-12-01T11:16:36.8087966Z   retention-days: 14
2022-12-01T11:16:36.8088240Z   if-no-files-found: ignore
2022-12-01T11:16:36.8088497Z   path: usage-log-*.zip
2022-12-01T11:16:36.8088748Z   name: artifact
2022-12-01T11:16:36.8089000Z   s3-bucket: gha-artifacts
2022-12-01T11:16:36.8089248Z   region: us-east-1
2022-12-01T11:16:36.8089478Z env:
2022-12-01T11:16:36.8089717Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:36.8089965Z   GPU_FLAG: --gpus all
2022-12-01T11:16:36.8090213Z ##[endgroup]
2022-12-01T11:16:37.2476762Z NOTE: s3-prefix specified, ignoring name parameter
2022-12-01T11:16:37.2477588Z With the provided path, there will be 1 file uploaded
2022-12-01T11:16:37.2477968Z Uploading to s3 prefix: pytorch/pytorch/3591403534/1/artifact
2022-12-01T11:16:37.2488917Z Starting upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304.zip
2022-12-01T11:16:37.4304875Z Finished upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_9818608304.zip
2022-12-01T11:16:37.4464869Z ##[group]Run set -x
2022-12-01T11:16:37.4465176Z set -x
2022-12-01T11:16:37.4465461Z python3 -m pip install -r requirements.txt
2022-12-01T11:16:37.4465804Z python3 -m pip install boto3==1.19.12
2022-12-01T11:16:37.4466202Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-12-01T11:16:37.4480093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:37.4480402Z env:
2022-12-01T11:16:37.4480646Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:37.4480899Z   GPU_FLAG: --gpus all
2022-12-01T11:16:37.4481171Z   AWS_DEFAULT_REGION: us-east-1
2022-12-01T11:16:37.4481441Z   BRANCH: pull/89997
2022-12-01T11:16:37.4481686Z   TEST_CONFIG: distributed
2022-12-01T11:16:37.4481945Z   SHARD_NUMBER: 1
2022-12-01T11:16:37.4482263Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T11:16:37.4482568Z   PR_NUMBER: 89997
2022-12-01T11:16:37.4482970Z   PYTORCH_RETRY_TEST_CASES: 1
2022-12-01T11:16:37.4483297Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2022-12-01T11:16:37.4483603Z   SHA1: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T11:16:37.4483879Z   TAG: 
2022-12-01T11:16:37.4484114Z   WORKFLOW_ID: 3591403534
2022-12-01T11:16:37.4484539Z   GITHUB_TOKEN: ***
2022-12-01T11:16:37.4484789Z   GHA_WORKFLOW_JOB_ID: 9818608304
2022-12-01T11:16:37.4485050Z ##[endgroup]
2022-12-01T11:16:37.4515554Z + python3 -m pip install -r requirements.txt
2022-12-01T11:16:37.7465920Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T11:16:37.7805019Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2022-12-01T11:16:37.7843365Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2022-12-01T11:16:37.7854670Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2022-12-01T11:16:37.7868493Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.58.2)
2022-12-01T11:16:37.8420459Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2022-12-01T11:16:37.8433039Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2022-12-01T11:16:37.8547838Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2022-12-01T11:16:37.8559480Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2022-12-01T11:16:37.8809974Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2022-12-01T11:16:37.9060478Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2022-12-01T11:16:37.9073316Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2022-12-01T11:16:37.9081831Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2022-12-01T11:16:37.9095728Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2022-12-01T11:16:37.9125884Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2022-12-01T11:16:37.9148747Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.0.4)
2022-12-01T11:16:37.9172510Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.1.0)
2022-12-01T11:16:37.9550652Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2022-12-01T11:16:37.9565712Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2022-12-01T11:16:37.9593006Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2022-12-01T11:16:37.9608984Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.13)
2022-12-01T11:16:37.9834437Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.9.24)
2022-12-01T11:16:37.9846361Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2022-12-01T11:16:38.0945027Z + python3 -m pip install boto3==1.19.12
2022-12-01T11:16:38.3868963Z Defaulting to user installation because normal site-packages is not writeable
2022-12-01T11:16:38.4097190Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2022-12-01T11:16:38.4167993Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2022-12-01T11:16:38.4185604Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2022-12-01T11:16:38.4224127Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2022-12-01T11:16:38.4293904Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.13)
2022-12-01T11:16:38.4518084Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2022-12-01T11:16:38.4546251Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2022-12-01T11:16:38.7051067Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2022-12-01T11:16:47.1068464Z [scribe] Scribe access token not provided, sending report via boto3...
2022-12-01T11:16:47.1068812Z 
2022-12-01T11:16:47.1069179Z ----- Historic stats comparison result ------
2022-12-01T11:16:47.1069836Z 
2022-12-01T11:16:47.1070103Z     job: linux-bionic-cuda11.6-py3.10-gcc7
2022-12-01T11:16:47.1070473Z     commit: c13d400bffe90e16b96520bbc8a41a6f0c9cd584
2022-12-01T11:16:47.1070683Z 
2022-12-01T11:16:47.1070897Z Commit graph (base is most recent master ancestor with at least one S3 report):
2022-12-01T11:16:47.1071153Z 
2022-12-01T11:16:47.1071257Z     : (master)
2022-12-01T11:16:47.1071465Z     |
2022-12-01T11:16:47.1071760Z     | * c13d400bff (HEAD)            total time  1084.49s
2022-12-01T11:16:47.1072032Z     | |
2022-12-01T11:16:47.1072237Z     | : (59 commits)
2022-12-01T11:16:47.1072468Z     |/
2022-12-01T11:16:47.1072882Z     * 67eb2d5952 (base)  18 reports, total time  3503.90s ± 2079.90s
2022-12-01T11:16:47.1073311Z     * 1c5ca724f4          9 reports, total time  3529.71s ± 2080.27s
2022-12-01T11:16:47.1073750Z     * 9d6109c4b0          9 reports, total time  3530.22s ± 2110.41s
2022-12-01T11:16:47.1074513Z     * 736adc0808          9 reports, total time  3501.80s ± 2137.81s
2022-12-01T11:16:47.1074980Z     * a348975e00          9 reports, total time  3551.46s ± 2118.11s
2022-12-01T11:16:47.1075385Z     * db13049b88          9 reports, total time  3520.57s ± 2066.49s
2022-12-01T11:16:47.1075810Z     * d07b85393a          9 reports, total time  3540.86s ± 2074.85s
2022-12-01T11:16:47.1076242Z     * ac25c210e5          9 reports, total time  3549.18s ± 2105.70s
2022-12-01T11:16:47.1076643Z     * 2355b6256b          9 reports, total time  3596.87s ± 2152.28s
2022-12-01T11:16:47.1079282Z     * 4f95f7ae9b          9 reports, total time  3534.00s ± 2151.51s
2022-12-01T11:16:47.1079593Z     |
2022-12-01T11:16:47.1079789Z     :
2022-12-01T11:16:47.1079929Z 
2022-12-01T11:16:47.1080102Z Removed  (across  960 suites)     0 tests, totaling     0.00s
2022-12-01T11:16:47.1080462Z Modified (across    0 suites)     0 tests, totaling     0.00s
2022-12-01T11:16:47.1081015Z Added    (across   17 suites)   371 tests, totaling +4998.06s
2022-12-01T11:16:47.1634470Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2022-12-01T11:16:47.1634812Z with:
2022-12-01T11:16:47.1635026Z env:
2022-12-01T11:16:47.1635265Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:47.1635514Z   GPU_FLAG: --gpus all
2022-12-01T11:16:47.1635758Z ##[endgroup]
2022-12-01T11:16:47.1655180Z ##[group]Run set -eou pipefail
2022-12-01T11:16:47.1655496Z set -eou pipefail
2022-12-01T11:16:47.1655746Z 
2022-12-01T11:16:47.1656067Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2022-12-01T11:16:47.1656392Z for _ in $(seq 1440); do
2022-12-01T11:16:47.1656690Z     # Break if no ssh session exists anymore
2022-12-01T11:16:47.1656990Z     if [ "$(who)" = "" ]; then
2022-12-01T11:16:47.1657224Z       break
2022-12-01T11:16:47.1657459Z     fi
2022-12-01T11:16:47.1657690Z     echo "."
2022-12-01T11:16:47.1657921Z     sleep 5
2022-12-01T11:16:47.1658192Z done
2022-12-01T11:16:47.1671961Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:47.1672250Z env:
2022-12-01T11:16:47.1672497Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:47.1672766Z   GPU_FLAG: --gpus all
2022-12-01T11:16:47.1672996Z ##[endgroup]
2022-12-01T11:16:47.1702635Z Holding runner for 2 hours until all ssh sessions have logged out
2022-12-01T11:16:47.1753815Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2022-12-01T11:16:47.1754250Z # ignore expansion of "docker ps -q" since it could be empty
2022-12-01T11:16:47.1754597Z # shellcheck disable=SC2046
2022-12-01T11:16:47.1754911Z docker stop $(docker ps -q) || true
2022-12-01T11:16:47.1755208Z # Prune all of the docker images
2022-12-01T11:16:47.1755759Z docker system prune -af
2022-12-01T11:16:47.1768349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-12-01T11:16:47.1768640Z env:
2022-12-01T11:16:47.1768892Z   GIT_DEFAULT_BRANCH: master
2022-12-01T11:16:47.1769162Z   GPU_FLAG: --gpus all
2022-12-01T11:16:47.1769392Z ##[endgroup]
2022-12-01T11:16:47.5796563Z 7ff5451e42d4
2022-12-01T11:16:48.3373218Z Deleted Containers:
2022-12-01T11:16:48.3373619Z 7ff5451e42d4638cf6b6098bb9ca823aa5cb3bfb8c9e9a1aea66c1e90c6dce30
2022-12-01T11:16:48.3373878Z 
2022-12-01T11:16:53.3786586Z Deleted Images:
2022-12-01T11:16:53.3787487Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7:fa72f5a0a230eb632055220542038bd4ceca184b
2022-12-01T11:16:53.3788482Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.6-cudnn8-py3-gcc7@sha256:217fd7de680e1dd5bca4b2b4054bd05a8d454df5e210ffbf1e5955e01cf1f340
2022-12-01T11:16:53.3789129Z deleted: sha256:f1c05e1c4a24b2f30e4b2dd175e18efaf97173a54d9813ad2e6088a740690342
2022-12-01T11:16:53.3790011Z deleted: sha256:1e7def21582cf03150a2e3ad04e2701388eaa422b3720fe7f648241a8988388e
2022-12-01T11:16:53.3790459Z deleted: sha256:3a7e72ba714442f54696c557b74dca88c618446caa68ff6506c2ce93cd7b2a85
2022-12-01T11:16:53.3790905Z deleted: sha256:acc47a47b0a4383dfbd2bd91be28c265a6486f388812591a0a9c185c62047c0a
2022-12-01T11:16:53.3791364Z deleted: sha256:60f22f4a0a4afdbef80bed81f20ae4c7a54233808a9544253a3c30a2776d2f71
2022-12-01T11:16:53.3791805Z deleted: sha256:5efa4324ff86638bcfa35ab9b1debaddf3d7513732c27ca8dd7e60e8358da6c9
2022-12-01T11:16:53.3792257Z deleted: sha256:a7f286562f5d9f9a1490bbc0a8125ba19aefca94af08516be2493eeb191f127f
2022-12-01T11:16:53.3792700Z deleted: sha256:e403df4fa1d82e23b8db68b6e2b5562fcd55535dd72214b9421e7814eee51ffd
2022-12-01T11:16:53.3793139Z deleted: sha256:f9471dd8edb7c2e1eb3aefda26ec6e67ee49347262f97f3111f176eeb2bf9465
2022-12-01T11:16:53.3793593Z deleted: sha256:d6df7f8a51a6e7483b61b1f330f3a4417b815fce5a7ad4c2a89510a54f5537da
2022-12-01T11:16:53.3794022Z deleted: sha256:b12c66528bc64f807b7a52116ecdf950025b66a403dd328a33db5511b63fc3f1
2022-12-01T11:16:53.3794762Z deleted: sha256:e8a84d179a24426205a63771d9aa894a6889147decc6d227312415e2842d0e38
2022-12-01T11:16:53.3795178Z deleted: sha256:b1e802e06103bb69d2a413954707d2f281a894d36a1f7fa3a65efa626a0e5e43
2022-12-01T11:16:53.3795609Z deleted: sha256:8efc9a5c3ed0496241d3ce1f76339b4d1270b5470ce1429effc703ca61a04c50
2022-12-01T11:16:53.3796078Z deleted: sha256:fef4f8c199945077358334d285e9dbe762a116427ec29d567139bee51863bef4
2022-12-01T11:16:53.3796513Z deleted: sha256:3f9f9d5fa745067d0ddf7f1206fcf51b87304adb8d32c87e130cbaae4acee275
2022-12-01T11:16:53.3796937Z deleted: sha256:75fb962b88e95b55a260c8cebb5da299635f32f3b104d08b08a2e1fba788437b
2022-12-01T11:16:53.3797377Z deleted: sha256:bd2c58efdd30f071840e8b3fd569126cf238bfdd5f0cd18345e36b60f93489df
2022-12-01T11:16:53.3797837Z deleted: sha256:d72aae1f9d4b7e1acc7b1954fed6c4886c49cec893a48dac5337fee3c9bf4c08
2022-12-01T11:16:53.3798289Z deleted: sha256:2b0461664863c4ef4ad02da7c9ec92a1d6d08bc8a9b76b5a982b7a98549980d3
2022-12-01T11:16:53.3798703Z deleted: sha256:086197a6cf735060ac5a6e595333f0bc14643bfddd3c338439c22002c1bec257
2022-12-01T11:16:53.3799113Z deleted: sha256:64254e4d362f949e9732531c579ca041aa47d4f70f7c242fda19048d82df0a1d
2022-12-01T11:16:53.3799535Z deleted: sha256:25374ed655bf3ce154d4ee14bd8c4f3220431d67b0d18acf3efdd666bbc01285
2022-12-01T11:16:53.3799959Z deleted: sha256:70a8b1a97ab5caea7ba2d2ecea43b58781672c588ee6375271d2b5f04d31594c
2022-12-01T11:16:53.3800404Z deleted: sha256:758af05d13f6edc84594db4db15f7b8066c5da10f6d83717a1d8b6ae5a2caccf
2022-12-01T11:16:53.3800833Z deleted: sha256:b37c30a9368604ae9d1873491a752ae71aba89de73dec5588638c1d2e3917901
2022-12-01T11:16:53.3801229Z deleted: sha256:790f109bf6ce127a4f3e25ea1675387993204350100fca6cecb56f8b863ce2e5
2022-12-01T11:16:53.3801667Z deleted: sha256:df13ec8c62d036e4b9fe0fd768e7fae7c5f76430b4aae4f3f356d701621b86d8
2022-12-01T11:16:53.3802111Z deleted: sha256:ef027ba5732bd4bc180cc8f8d769cd28630e92609a88be8f472e4c8ee8a3bb6d
2022-12-01T11:16:53.3802542Z deleted: sha256:5751424dd4ee3deaaf3e9f8b8172683ff7f30d90dc2415c3833042333f52b619
2022-12-01T11:16:53.3802953Z deleted: sha256:ac0364f6b4441a779e12c1527f256b052fff92b14aca383c87e62a8d6e4d1470
2022-12-01T11:16:53.3803364Z deleted: sha256:1f8161738ca37b8a92ed2577a6d9705167fb325ad8f8512439b47251013af048
2022-12-01T11:16:53.3803792Z deleted: sha256:2f8770a5227a4d66f9bdea07ca3e6e7cda5b06fbc362b354ae29f7542b3e7a19
2022-12-01T11:16:53.3804211Z deleted: sha256:652470cd61eae170df59d88a3b038cd2ab199526f4ba54badf85dd5628d7cb37
2022-12-01T11:16:53.3804655Z deleted: sha256:155eec0e8b217d2bafa06b3f85e987c3164bbad427de092045ddbff33f091a1f
2022-12-01T11:16:53.3805126Z deleted: sha256:8c60c1d76e4c0216ef0cc9248e31a656ab0a29bc95f589f4b23a0634523058c7
2022-12-01T11:16:53.3805540Z deleted: sha256:2811ebe06c45b7997659f3d50562406d001f3e979f398f9b4844bac84b3c419e
2022-12-01T11:16:53.3805949Z deleted: sha256:6a690d8e14cafcab7ab578a42baf2ca075033cf306f03e13ae70150b8be2e586
2022-12-01T11:16:53.3806396Z deleted: sha256:635a5f6faf92c02fc91f0a365e65bb840b74b3b7c079ee1009e6c15cd7c0e7b4
2022-12-01T11:16:53.3806843Z deleted: sha256:2696ecc06a0fc00b7ef534ef78758030c9d69f082f8fcb2f5ab275c6fc199e7c
2022-12-01T11:16:53.3807275Z deleted: sha256:cec67903bb9f9cc16c8826dff0d76c876d88147fecd2dec21ebc0adc56e4f149
2022-12-01T11:16:53.3807709Z deleted: sha256:6d95f163e75e744dc79893bc297644cf2a70187540d68c847baeb77483c6a0f5
2022-12-01T11:16:53.3808144Z deleted: sha256:1b1204e913bbc6e599fc1cdcabede5cd1315dc1b55cd67052bc665a8f8e3cc1a
2022-12-01T11:16:53.3808583Z deleted: sha256:93670f5e92be335224cf3a8b69a526306d2fec5e5e4e42c272b8c0458060036a
2022-12-01T11:16:53.3808993Z deleted: sha256:a0b1388f987804a153c13745107eb2a30de9cc6dc9eab87ae191707765dd7175
2022-12-01T11:16:53.3809413Z deleted: sha256:129bdb873e79117f4e90135f0c6a58f775fcf596f4eb514b803771cef2da8278
2022-12-01T11:16:53.3809862Z deleted: sha256:2d49e3a81bd436bfd20fb4a849cdc98da82cb74afef3de38dda7a946d3fc4153
2022-12-01T11:16:53.3810296Z deleted: sha256:0ba4e259108e5311ddf6b79ae3a35f8f16a4004ef8817e50427baa3cc90ac081
2022-12-01T11:16:53.3810872Z deleted: sha256:c164403226561914f16becdeca65c54d20dba8dad414b062efc34c05c47bf725
2022-12-01T11:16:53.3811419Z deleted: sha256:cbe4006b2e6286d50c1b292fb71b69d5299d65f055285519eafc41eac3ef8a3c
2022-12-01T11:16:53.3811855Z deleted: sha256:edcec18dceb25f1a03ec20de4676464613e69072875a83f5c45e45a31aafc5b9
2022-12-01T11:16:53.3812269Z deleted: sha256:13c4f317ac4bb48997302756b8d5f8b602e835607c9806a1a5b200e9a0657d8a
2022-12-01T11:16:53.3812675Z deleted: sha256:57f043e380f4586c76968d6e062b50bac55254a5be7e80bea3c027a5bb316469
2022-12-01T11:16:53.3813093Z deleted: sha256:3e549931e0240b9aac25dc79ed6a6259863879a5c9bd20755f77cac27c1ab8c8
2022-12-01T11:16:53.3813330Z 
2022-12-01T11:16:53.3813473Z Total reclaimed space: 18.87GB
2022-12-01T11:16:53.3876225Z Post job cleanup.
2022-12-01T11:16:53.3915500Z Post job cleanup.
2022-12-01T11:16:53.5301091Z [command]/usr/bin/git version
2022-12-01T11:16:53.5356813Z git version 2.37.1
2022-12-01T11:16:53.5420849Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/81a5ff1d-f6d8-4962-acca-cda97b794b63' before making global git config changes
2022-12-01T11:16:53.5421450Z Adding repository directory to the temporary git global config as a safe directory
2022-12-01T11:16:53.5427474Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2022-12-01T11:16:53.5472206Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-12-01T11:16:53.5508770Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-12-01T11:16:53.5836888Z Entering 'android/libs/fbjni'
2022-12-01T11:16:53.5880163Z Entering 'third_party/FP16'
2022-12-01T11:16:53.5923176Z Entering 'third_party/FXdiv'
2022-12-01T11:16:53.5968496Z Entering 'third_party/NNPACK'
2022-12-01T11:16:53.6014714Z Entering 'third_party/QNNPACK'
2022-12-01T11:16:53.6060561Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T11:16:53.6107663Z Entering 'third_party/XNNPACK'
2022-12-01T11:16:53.6165714Z Entering 'third_party/benchmark'
2022-12-01T11:16:53.6210332Z Entering 'third_party/cpuinfo'
2022-12-01T11:16:53.6260231Z Entering 'third_party/cub'
2022-12-01T11:16:53.6308598Z Entering 'third_party/cudnn_frontend'
2022-12-01T11:16:53.6363402Z Entering 'third_party/cutlass'
2022-12-01T11:16:53.6419271Z Entering 'third_party/eigen'
2022-12-01T11:16:53.6467064Z Entering 'third_party/fbgemm'
2022-12-01T11:16:53.6521852Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T11:16:53.6568368Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T11:16:53.6611830Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T11:16:53.6655106Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T11:16:53.6699785Z Entering 'third_party/flatbuffers'
2022-12-01T11:16:53.6747592Z Entering 'third_party/fmt'
2022-12-01T11:16:53.6794649Z Entering 'third_party/foxi'
2022-12-01T11:16:53.6837606Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T11:16:53.6883561Z Entering 'third_party/gloo'
2022-12-01T11:16:53.6928227Z Entering 'third_party/googletest'
2022-12-01T11:16:53.6973521Z Entering 'third_party/ideep'
2022-12-01T11:16:53.7018212Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T11:16:53.7062719Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T11:16:53.7114850Z Entering 'third_party/ios-cmake'
2022-12-01T11:16:53.7159403Z Entering 'third_party/ittapi'
2022-12-01T11:16:53.7204880Z Entering 'third_party/kineto'
2022-12-01T11:16:53.7249484Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T11:16:53.7294000Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T11:16:53.7341005Z Entering 'third_party/nccl/nccl'
2022-12-01T11:16:53.7386611Z Entering 'third_party/neon2sse'
2022-12-01T11:16:53.7431898Z Entering 'third_party/nlohmann'
2022-12-01T11:16:53.7480004Z Entering 'third_party/onnx'
2022-12-01T11:16:53.7539744Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T11:16:53.7583859Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T11:16:53.7632353Z Entering 'third_party/onnx-tensorrt'
2022-12-01T11:16:53.7677294Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T11:16:53.7725221Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T11:16:53.7768102Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T11:16:53.7812040Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T11:16:53.7860737Z Entering 'third_party/pocketfft'
2022-12-01T11:16:53.7905390Z Entering 'third_party/protobuf'
2022-12-01T11:16:53.7955012Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T11:16:53.7999457Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T11:16:53.8044647Z Entering 'third_party/psimd'
2022-12-01T11:16:53.8089278Z Entering 'third_party/pthreadpool'
2022-12-01T11:16:53.8133163Z Entering 'third_party/pybind11'
2022-12-01T11:16:53.8179995Z Entering 'third_party/python-enum'
2022-12-01T11:16:53.8228334Z Entering 'third_party/python-peachpy'
2022-12-01T11:16:53.8276938Z Entering 'third_party/python-six'
2022-12-01T11:16:53.8324848Z Entering 'third_party/sleef'
2022-12-01T11:16:53.8373872Z Entering 'third_party/tbb'
2022-12-01T11:16:53.8425843Z Entering 'third_party/tensorpipe'
2022-12-01T11:16:53.8471862Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T11:16:53.8514291Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T11:16:53.8557828Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T11:16:53.8600942Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T11:16:53.8643780Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T11:16:53.8691903Z Entering 'third_party/zstd'
2022-12-01T11:16:53.8756099Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-12-01T11:16:53.8787771Z http.https://github.com/.extraheader
2022-12-01T11:16:53.8798023Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-12-01T11:16:53.8836408Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-12-01T11:16:53.9172993Z Entering 'android/libs/fbjni'
2022-12-01T11:16:53.9199722Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9234346Z Entering 'third_party/FP16'
2022-12-01T11:16:53.9259203Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9293726Z Entering 'third_party/FXdiv'
2022-12-01T11:16:53.9319944Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9354537Z Entering 'third_party/NNPACK'
2022-12-01T11:16:53.9379758Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9414302Z Entering 'third_party/QNNPACK'
2022-12-01T11:16:53.9440795Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9474634Z Entering 'third_party/VulkanMemoryAllocator'
2022-12-01T11:16:53.9499691Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9533586Z Entering 'third_party/XNNPACK'
2022-12-01T11:16:53.9560197Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9605525Z Entering 'third_party/benchmark'
2022-12-01T11:16:53.9631381Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9664664Z Entering 'third_party/cpuinfo'
2022-12-01T11:16:53.9691256Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9725632Z Entering 'third_party/cub'
2022-12-01T11:16:53.9751526Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9784733Z Entering 'third_party/cudnn_frontend'
2022-12-01T11:16:53.9810692Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9850330Z Entering 'third_party/cutlass'
2022-12-01T11:16:53.9875850Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9918202Z Entering 'third_party/eigen'
2022-12-01T11:16:53.9943076Z http.https://github.com/.extraheader
2022-12-01T11:16:53.9978956Z Entering 'third_party/fbgemm'
2022-12-01T11:16:54.0004960Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0039673Z Entering 'third_party/fbgemm/third_party/asmjit'
2022-12-01T11:16:54.0064213Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0097311Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2022-12-01T11:16:54.0123555Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0160812Z Entering 'third_party/fbgemm/third_party/googletest'
2022-12-01T11:16:54.0184780Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0219938Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2022-12-01T11:16:54.0246435Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0281610Z Entering 'third_party/flatbuffers'
2022-12-01T11:16:54.0307474Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0343862Z Entering 'third_party/fmt'
2022-12-01T11:16:54.0369891Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0408969Z Entering 'third_party/foxi'
2022-12-01T11:16:54.0434437Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0467286Z Entering 'third_party/gemmlowp/gemmlowp'
2022-12-01T11:16:54.0494143Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0528290Z Entering 'third_party/gloo'
2022-12-01T11:16:54.0554280Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0587553Z Entering 'third_party/googletest'
2022-12-01T11:16:54.0613700Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0647144Z Entering 'third_party/ideep'
2022-12-01T11:16:54.0672748Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0704779Z Entering 'third_party/ideep/mkl-dnn'
2022-12-01T11:16:54.0729820Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0764720Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2022-12-01T11:16:54.0790332Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0832412Z Entering 'third_party/ios-cmake'
2022-12-01T11:16:54.0857511Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0891832Z Entering 'third_party/ittapi'
2022-12-01T11:16:54.0918035Z http.https://github.com/.extraheader
2022-12-01T11:16:54.0951088Z Entering 'third_party/kineto'
2022-12-01T11:16:54.0975809Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1009720Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2022-12-01T11:16:54.1034654Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1067844Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2022-12-01T11:16:54.1095383Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1130673Z Entering 'third_party/nccl/nccl'
2022-12-01T11:16:54.1156899Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1190021Z Entering 'third_party/neon2sse'
2022-12-01T11:16:54.1215791Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1250241Z Entering 'third_party/nlohmann'
2022-12-01T11:16:54.1275281Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1310964Z Entering 'third_party/onnx'
2022-12-01T11:16:54.1335580Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1382737Z Entering 'third_party/onnx/third_party/benchmark'
2022-12-01T11:16:54.1408122Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1441933Z Entering 'third_party/onnx/third_party/pybind11'
2022-12-01T11:16:54.1466249Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1502686Z Entering 'third_party/onnx-tensorrt'
2022-12-01T11:16:54.1529623Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1601803Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2022-12-01T11:16:54.1625573Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1663604Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2022-12-01T11:16:54.1689301Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1722133Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2022-12-01T11:16:54.1746703Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1795801Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2022-12-01T11:16:54.1820412Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1859131Z Entering 'third_party/pocketfft'
2022-12-01T11:16:54.1885705Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1920688Z Entering 'third_party/protobuf'
2022-12-01T11:16:54.1945323Z http.https://github.com/.extraheader
2022-12-01T11:16:54.1982366Z Entering 'third_party/protobuf/third_party/benchmark'
2022-12-01T11:16:54.2007121Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2040318Z Entering 'third_party/protobuf/third_party/googletest'
2022-12-01T11:16:54.2064775Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2100163Z Entering 'third_party/psimd'
2022-12-01T11:16:54.2125809Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2160297Z Entering 'third_party/pthreadpool'
2022-12-01T11:16:54.2184528Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2217079Z Entering 'third_party/pybind11'
2022-12-01T11:16:54.2241936Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2275341Z Entering 'third_party/python-enum'
2022-12-01T11:16:54.2299881Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2332438Z Entering 'third_party/python-peachpy'
2022-12-01T11:16:54.2358037Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2390016Z Entering 'third_party/python-six'
2022-12-01T11:16:54.2414544Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2447406Z Entering 'third_party/sleef'
2022-12-01T11:16:54.2472548Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2505974Z Entering 'third_party/tbb'
2022-12-01T11:16:54.2530873Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2577556Z Entering 'third_party/tensorpipe'
2022-12-01T11:16:54.2602335Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2635524Z Entering 'third_party/tensorpipe/third_party/googletest'
2022-12-01T11:16:54.2659323Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2692993Z Entering 'third_party/tensorpipe/third_party/libnop'
2022-12-01T11:16:54.2717365Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2749070Z Entering 'third_party/tensorpipe/third_party/libuv'
2022-12-01T11:16:54.2774043Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2808191Z Entering 'third_party/tensorpipe/third_party/pybind11'
2022-12-01T11:16:54.2833098Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2865449Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2022-12-01T11:16:54.2890944Z http.https://github.com/.extraheader
2022-12-01T11:16:54.2926725Z Entering 'third_party/zstd'
2022-12-01T11:16:54.2952074Z http.https://github.com/.extraheader
2022-12-01T11:16:54.3266503Z Cleaning up orphan processes